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and  analyze  tools  and  procedures  which  facilitate  the  ISO  process.  Findings  of 
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duced by  the  civilian  environment;  and  (d)  more  procedures  are  available  than 
either  tools  or  methodologies.  Approximately  sixty  authoring  tools  and  proce- 
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SUMMARY 

PROBLEM 

The  authoring  process  and  authoring  aids  which  facilitate  development  of 
instructional  materials  has  recently  emerged  as  an  area  of  concern  in  the  field 
of  Instructional  Systems  Development  (ISD).  This  is  due  to  a growing  realization 
that  instructional  materials  development  is  a major  cost  factor  in  developing 
instructional  systems.  The  purpose  of  this  study  was  to  examine  the  ISD  concept 
as  it  relates  to  the  authoring  process,  and  to  survey,  identify,  and  analyze 
authoring  tools  and  procedures  which  aid  the  ISD  process. 

APPROACH 

Both  American  and  Canadian  sources  from  1965  to  the  present  were  searched  in 
the  following  areas:  (a)  developmental  studies  of  on-  and  off-line  instructional 

systems;  (b)  evaluation  and  comparison  of  developmental  methodologies  for  instruc- 
tional materials;  (c)  design  and  installation  of  instructional  materials;  (d)  "user 
studies"  involving  effectiveness  of  a medium  and  comparative  effectiveness  of  alter- 
native media;  (e)  information  processing  behavior  of  students;  (f)  analyses  of 
training  tasks  and  job  requirements;  and  (g)  revision  procedures  for  instructional 
materials  and  systems,  both  on-  and  off-line. 

Computerized  searches  were  made  of  data  banks  in  the  Educational  Resources 
Information  Center  (ERIC),  the  National  Technical  Information  Service  (NTIS),  and 
the  libraries  of  McDonnell  Douglas  Corporation.  Manual  searches  were  made  of 
indexes,  abstracting  services,  and  bibliographies  at  several  local  universities. 

A Selective  Dissemination  of  Information  (SDI)  service  was  used  along  with  personal 
communication  with  over  seventy  individuals  and  associations  prominent  in  the  field 
of  instructional  technology. 

In  total,  over  five  thousand  reports  were  read  in  full-text  or  abstract  form, 
and  approximately  fourteen  hundred  practical , development-oriented  documents  were 
selected  for  inclusion  in  a computer-based  information  storage  and  retrieval  system 
developed  for  the  study.  The  documents  were  categorized  by:  (a)  author;  (b)  insti- 

tution; (c)  title;  (d)  publication  date;  (e)  applicable  ISD  step;  (f)  type  of  media; 
(g)  type  of  application;  (h)  type  of  evaluation;  and  (i)  type  of  environment. 
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RESULTS 

Findings  of  the  study  include:  (a)  more  applied  but  unevaluated  documents  are 

associated  with  the  civilian  environment;  (b)  relatively  equal  numbers  of  applied 
and  evaluated  documents  exist  in  both  civilian  and  military  environments;  (c)  a 
greater  total  number  of  authoring  aids  (tools,  procedures,  and  methodologies)  tends 
to  be  produced  by  the  civilian  environment;  and  (d)  more  procedures  are  available 
than  either  tools  or  methodologies.  Six  significant  aspects  of  the  authoring 
process  were  also  identified. 

In  addition,  an  extensive  annotated  bibliography  of  approximately  sixty 
authoring  tools  and  procedures  was  produced.  Based  on  the  Inter-service  Procedures 
for  Instructional  Systems  Development  (IPISD)  Model,  the  annotated  bibliography 
groups  documents  by  ISD  phase  and  step,  and  provides  citations,  abstracts,  and 
analytic  information. 

CONCLUSIONS 

Recommendations  are  made  for  creating  tools  for  ISD  steps  which  have  none,  for 
refining  procedures  into  tools  for  ISD  steps  where  tools  are  more  useful,  and  for 
developing  methodologies  for  ISD  steps  where  the  state-of-the-art  is  in  advance  of 
current  tools  and  procedures. 
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1.0  INTRODUCTION 


1.1  BACKGROUND 

1.1.1  Instructional  System  Development  (ISP)  - Instructional  Systems  Develop- 
ment is  an  adaptation  of  the  systems  approach,  a sequence  of  steps  by  which  complex 
systems  are  broken  down  into  smaller,  more  manageable  problem  areas  that  can  then 
be  analyzed  for  use  in  analysis,  design,  development,  implementation,  and  control 
of  an  instructional  system.  The  systems  approach  has  been  adapted  to  instruction 
in  order  to  define  such  things  as  job  activities,  testing,  sequencing  instruction, 
and  evaluating  instructional  systems  in  terms  of  both  learners  and  organizational 
as  well  as  physical  components. 

The  Interservice  Procedures  for  Instructional  Systems  Development  (IPISD) 
provided  the  ISD  model  used  in  this  study  (TRADCC,  1975).  The  IPISD  model  consists 
of  five  phases  and  nineteen  steps.  Each  step  has  both  an  input  and  output.  The 
phases  and  steps  in  the  IPISD  model  are  not  necessarily  done  on  a once-and-once- 
only  basis  but  rather  provide  for  a continuing,  iterative  process  of  system 
development. 

1.1.2  Authoring  Tools  and  Procedures  - Authoring  tools  and  procedures  were 
identified  primarily  to  supply  information  for  decision-making  purposes,  not  to 
assess  their  merit.  Since  the  authoring  process  is  an  ISD  activity,  the  terms 
"authoring  tools"  and  "authoring  procedures"  must  be  defined  in  relation  to  an 
ISD  methodology.  An  ISD  methodology  is  a general  systems  approach  with  multiple 
components  which,  given  a certain  set  of  constraints,  is  used  to  produce  an  instruc 
tional  system.  An  authoring  procedure  is  a step  or  series  of  steps  which  operation 
ally  defines  one  component  of  the  methodology.  An  authoring  tool  is  an  evaluated, 
self-contained  product  in  a suitable  format  which  is  applied  to  the  analysis, 
design,  development,  implementation,  or  control  of  one  or  more  steps  of  a procedure 

The  authoring  process  is  conceived  as  a series  of  separate,  though  related, 
activities: 

0 Information  Gathering  - the  acquisition  of  supportive  information  from 
task  analysis,  existing  training  materials,  instructors,  and  incoming 
students  ^ ^ 

mteooMmmaj.  noumt^m  Mmvmonuumom  eoMMw*'.  m^mr 
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0 Information  Conversion  - the  development  of  that  information  into  properly 
sequenced  packages  with  format  and  structure  that  enable  learning  to  occur 
effectively  and  efficiently 

0 Information  Revision  - the  revision,  as  necessary,  of  the  information 
packages  during  formative  and  summative  evaluation  based  on  feedback  from 
students,  instructors,  and  administrators 

0 Information  Publication  - the  publication  of  those  packages  in  a con- 
struction which  meets  the  requirements  of  the  learner  and  the  training 
environment,  e.g.,  a lesson  intended  for  homework  may  require  a different 
construction  than  one  intended  for  in-class  presentation. 

1.2  PURPOSE  AND  SCOPE 

The  purpose  of  this  literature  search  was  to  identify  and  analyze  authoring 
tools  and  procedures  for  developing  instructional  materials  and  to  identify  infor- 
mation clearing  houses  and  data  banks  for  existing  instructional  materials.  Both 
American  and  Canadian  sources  from  1965  to  the  present  concerning  ISO  were  searched 
in  the  following  areas: 

0 Developmental  studies  of  on-  and  off-line  instructional  systems 

0 Evaluation  and  comparison  of  development  methodologies  for  instructional 
materials 

0 Design  and  installation  of  instructional  materials 

0 "User  studies"  involving  effectiveness  of  a medium  and  comparative 
effectiveness  of  alternative  media 

0 Information  processing  behavior  of  students 

0 Analyses  of  training  tasks  and  job  requirements 

0 Revision  procedures  for  instructional  materials  and  systems,  both  on- 
and  off-line. 
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1.3  APPROACH 

Computerized  searches  were  made  of  data  banks  in  the  Educational  Resources 
Information  Center  (ERIC),  the  National  Technical  Information  Service  (NTIS),  the 
Defense  Documentation  Center  (DDC),  and  the  lioraries  of  the  McDonnell  Douglas 
Corporation.  Manual  searches  were  made  of  indexes,  abstracting  services,  and 
bibliographies  at  several  area  universities.  A Selective  Dissemination  of  Infor- 
mation (SDI)  service  was  used  along  with  personal  communication  with  over  seventy 
individuals  and  associations  prominent  in  the  field  of  instructional  technology. 

In  total,  over  five  thousand  reports  were  read  in  full-text  or  abstract  form,  and 
approximately  fourteen  hundred  practical,  development-oriented  items  were  selected 
for  inclusion  in  the  computer-based  information  storage  and  retrieval  system 
developed  for  the  present  study.  Appendix  A contains  the  codes  whereby  reports  and 
other  documents  were  described  and  entered  in  the  data  base. 

1.4  RESULTS 

Six  significant  aspects  of  the  authoring  process  were  identified:  (1)  size 

and  configuration  of  authoring  teams;  (2)  knowledge  and  proficiency  of  course 
authors;  (3)  existing  course  structures,  lesson  materials,  and  course  control 
documents;  (4)  type  of  media  and  type  of  instruction;  (5)  type  and  extent  of 
evaluation  and  revision;  and  (6)  text-handling  and  materials  reproduction  systems. 
Since  the  only  subjective  aspect  of  the  authoring  process  is  knowledge  and  profi- 
ciency of  course  authors,  identification  and  distribution  of  authoring  tools  and 
procedures  would  allow  course  authors  to  develop  materials  from  a baseline  of  equal 
knowledge.  Thus,  the  other  objective  aspects  of  the  authoring  process  could  be 
categorized  and  manipulated  in  a "laboratory"  (i.e.,  controlled)  setting,  and  it 
should  be  possible  to  model  the  authoring  process  variables  and  to  formulate 
decision  rules  for  managing  the  materials  development  cycle  with  an  optimum  mix 
of  personnel  and  organization. 

It  was  found  that  at  least  one  authoring  aid  (tool,  procedure,  or  methodology) 
was  available  for  every  ISD  activity.  The  types  of  authoring  aids,  however,  are 
unevenly  distributed  among  ISO  activities.  Relatively  equal  numbers  of  authoring 
aids  are  produced  by  civilian  and,  military  environments.  Finally,  it  was  found 
that  more  procedures  than  either  tools  or  methodologies  are  available  to  analyze, 
design,  develop,  implement,  and  control  the  ISD  process. 
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An  annotated  bibliography  of  authoring  aids  for  developing  course  materials 
using  the  ISD  approach  was  prepared  as  Appendix  B.  It  includes  an  analysis  of 
documentation  related  to  ISD  and  the  identification  of  information  clearinghouses 
for  existing  instructional  materials.  The  bibliography  is  in  a format  ready  for 
use. 
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2.0  APPROACH 


2.1  PROBLEMS 

Although  the  national  investment  in  computer-based  educational  training  to 
date  is  some  $2  billion,  the  extensive  use  of  computer-related  instructional  tech- 
nology is  yet  to  be  realized.  Anastasio  and  Morgan  (1972),  the  Carnegie  Commission 
(1972),  Demerath  and  Daniels  (1973),  Levien  (1972),  and  Wilcox  (cited  in  Hunter, 
Kastner,  Rubin  and  Seidel,  1975)  attempted  to  determine  why  computer-related  tech- 
nology has  not  been  widely  adopted  for  instruction.  Hunter  et  al . (1975)  summarized 
their  findings  as  follows: 

a.  Materials  - High-quality  materials  and  materials  using  different  instruc- 
tional techniques  or  strategies  or  addressing  alternative  subjects  are  not 
readily  available. 

b.  Overlooked  Materials  - When  new  instructional  materials  are  developed, 
earlier  versions  of  the  work  are  automatically  considered  obsolete. 
Additionally,  instructional  materials  in  actual  use  are  often  overlooked. 

c.  Relationship  of  Materials  to  Curricula  - Lack  of  standardized  curricula 
prevents  a building-block  approach  to  and  comparative  evaluation  of 
different  sets  of  instructional  materials. 

d.  Documentation  - Clear,  extensive  documentation  of  innovative  projects  is 
seldom  available. 

e.  Attitudes  - Faculty  members  fear  role  reduction  and  dehumanization 
through  use  of  instructional  technology.  Administrators  feel  that 
computers,  media  carrels,  and  other  instructional  hardware  are  luxuries 
which  should  be  treated  separately  from  the  normal  cost  of  school  opera- 
tions and  therefore,  if  necessary,  can  be  dropped  from  the  budget  with 
little  harm  to  instruction  or  learning. 
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■f-  Skilled  Personnel  - Technically  competent  specialists  who  can  provide 
training  in  the  uses  and  techniques  of  Instructional  technology  are  in 
short  supply.  A corollary  is  that  broad  formal  training  in  instructional 
technology  is  not  generally  available. 

g.  Proof  of  Effectiveness  - There  are  few  examples  of  proven,  cost-effective 
computer-based  instructional  systems.  Coupled  with  the  scarcity  of 
evaluative  measures  applicable  to  computer-based  instruction,  there  is 

no  definitive  evidence  that  computers  improve  the  quality  of  learning. 

h.  Cost  - The  high  capital  investment  required  to  purchase  or  develop 
sophisticated  instructional  technology  is  a deterrent  to  use,  especially 
in  public  schools. 

Of  these  eight  problems,  four  are  concerned  with  instructional  materials,  the 
developmental  use  of  which  remains  typical  of  the  "cottage  industry"  approach 
criticized  by  Molnar: 

The  large  amount  of  uncoordinated  research  activities  and  the  lack  of 
preplanned  linkages  between  research  and  practice  has  led  to  the  exis- 
tence of  an  expensive  cottage  industry  in  educational  technology  which 
tends  to  retool  every  academic  year.  Researchers  and  educators  fre- 
quently demonstrate  a strong  resistance  to  the  use  of  someone  else's 
innovation.  It  has  been  said  that  if  there  was  a Nobel  prize  for 
educational  research,  we  would  have  to  nominate  an  entire  generation 
of  researchers  for  their  co-discovery  of  the  wheel.  (Molnar,  1971,  p.7) 

At  least  three  factors  have  forced  educational  researchers  to  develop  and 
apply  their  own  unique  methods  to  such  things  as  job  analysis,  test  generation, 
construction  of  behavioral  objectives,  and  implementation,  evaluation,  and  revision 
of  instruction.  First,  many  educators  feel  very  strongly  that  instruction  should 
have  a local,  indigenous  quality  (Demerath  & Daniels,  1973).  Second,  instructional 
development  efforts  are  usually  driven  by  a "raw  empiricism"  so  that 

Instructional  materials  are  prepared  on  the  basis  of  intuition,  folklore, 
or  experience  and  administered  to  members  of  the  target  population.  If 
the  students  pass  the  test,  the  product  is  considered  appropriate;  if  not, 
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the  materials  are  revised  and  tried  again.  This  tryout-revision  cycle 

is  repeated  until  the  product  works  or  the  developers  run  out  of 

resources  or  time.  (Merrill  and  Boutwell,  1973,  p.96) 

Third,  there  is  a lack  of  information  on  available  authoring  tools,  procedures, 
and  clearinghouses  for  existing  course  materials  (Logan,  1976). 

This  study  attempted  to  identify  and  evaluate  authoring  aids  and  information 
clearinghouses  that  facilitate  analyzing,  designing,  developing,  implementing,  and 
controlling  the  ISO  process.  In  addition,  four  research  hypotheses  were  tested; 

0 The  civilian  environment  has  a greater  number  of  applied,  unevaluated 
documents  than  the  military  environment 

0 The  civilian  environment  has  a greater  number  of  applied,  evaluated  docu- 
ments than  the  military  environment 

0 There  are  unequal  numbers  of  documents  within  the  three  authoring  and 
categories  (tools,  procedures,  and  methodologies) 

0 The  civilian  environment  has  a greater  number  of  authoring  aids  than  the 
military  environment. 

Authoring  tools  and  procedures  were  identified  primarily  to  supply  information 
for  decision-making  purposes,  not  to  assess  their  merit.  The  first  reason  for 
taking  this  approach  was  that  selection  criteria  did  not  include  evaluation  of  the 
relative  worth  of  a particular  tool  or  procedure.  Second,  data  resulting  from 
evaluation  of  a particular  tool  or  procedure  were  either  unavailable  or  not  clearly 
identified.  And  third,  the  study  emphasized  breadth,  not  depth,  and,  therefore, 
many  tools  and  procedures  in  as  many  ISD  activities  as  possible  were  identified  — 
regardless  of  their  relative  worth. 

The  terms  "authoring  tools"  and  "authoring  procedures"  are  defined  in  relation 
to  an  ISD  methodology.  Accordingly,  three  definitions  are  given.  An  ISO  methodology 
is  a general  systems  approach  with  multiple  components  which,  given  a certain  set 
of  constraints,  is  used  to  produce  an  instructional  system.  An  authoring  procedure 
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is  a step  or  series  of  steps  which  operationally  defines  one  component  of  the 
methodology.  An  authoring  tool  is  an  evaluated,  self-contained  product  in  a suit- 
able format  which  is  applied  to  the  analysis,  design,  development,  implementation, 
or  control  of  one  or  more  steps  of  a procedure.  Authoring  tools  and  procedures, 
of  course,  are  used  in  a materials  development  cycle  called  the  "authoring  process.' 

The  Interservice  Procedures  for  Instructional  Systems  Development  (IPISD) 
provided  the  model  for  ISD  used  in  this  study  (TRADOC,  1975).  The  complete  IPISD 
model  is  presented  in  Figure  1. 

2.2  METHOD 

2.2.1  Search  Strategy  - Both  American  and  Canadian  literature  from  1965  to 
the  present  were  searched  for  references  pertaining  to  instructional  systems  in  the 
following  areas: 

0 Developmental  studies  of  on-  and  off-line  instructional  systems 

0 Evaluation  and  comparison  of  instructional  materials  development 
methodologies 

0 Design  and  installation  of  instructional  materials 

0 "User  studies"  involving  effectiveness  of  a medium  and  comparative 
effectiveness  of  alternative  media 

0 Information  processing  behavior  of  students 

0 Analysis  of  training  tasks  and  job  requirements 

0 Procedures  for  revising  instructional  materials  and  systems,  both  on-line 
and  off-line. 


Key  words  used  in  the  literature 

0 Adaptive  Models 
0 Adjunct  Questions 
0 Advanced  Organizers 
0 Audio-visual  Materials 
0 Auto-Instructional  Aids 


search  were: 

0 Embedded  Questions 
0 Formative  Evaluation 
0 Instructional  Technology 
0 Job  Analysis 
0 Performance  Checks 
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FIGURE  1 IHTERSERVICE  PROCEDURES  FOR  INSTRUCTIONAL  SYSTEMS  DEVELOPMENT  MODEL 
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Computerized  searches  were  made  of  data  banks  in  the  Educational  Resources 
Information  Center  (ERIC),  the  National  Technical  Information  Service  (NTIS),  the 
Defense  Documentation  Center  (DDC),  and  the  libraries  of  the  McDonnell  Douglas 
Corporation.  Manual  searches  were  made  of  indexes,  abstracting  services,  and 
bibliographies  at  several  local  universities.  In  addition,  a Selective  Dissemin- 
ation of  Information  (SDI)  service  was  used  along  with  personal  communication  with 
over  seventy  individuals  and  associations  prominent  in  the  field  of  instructional 
technology.  Over  5,000  reports  were  read  in  full  text  or  abstract  form,  and 
approximately  1,400  practical,  development-oriented  items  were  selected  for 
inclusion  in  the  computerized  data  base  developed  for  the  study.  Approximately 
sixty  documents  were  selected  from  the  data  base  for  inclusion  in  the  annotated 
bibliography.  Appendix  B. 


2.2.2  Information  Storage  and  Retrieval  System  - A computer-based  information 
storage  and  retrieval  system  was  established  for  the  study.  Hardware  consisted  of 

a teletype  terminal  connected  via  a communications  link  to  a Xerox  Sigma  9 computer. 
Printouts  and  formal  reports  were  processed  at  a commercial  computer-service  office. 
Software  consisted  of  two,  high-level  English  language  "packaged"  programs:  EDIT 

and  CONFIRM.  EDIT  permitted  easy  data  entry  and  allowed  document  titles  to  be 
accessed  by  keyword.  CONFIRM  was  used  for  complex  retrieval  operations,  statistical 
counts,  and  report  generation. 

2.2.3  Selection  Criteria  - Authoring  aids  and  information  clearinghouses  were 
classified  as  either  tools,  procedures,  or  methodologies  used  to  analyze,  design, 
develop,  implement,  and  control  ISO  activities.  Research  reports  and  brochures 
from  clearinghouses  were  selected  for  inclusion  in  the  data  base  according  to  three 
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criteria.  First,  selected  documents  had  to  fit  the  definition  of  a tool,  procedure, 
or  methodology.  Second,  since  the  end  result  of  the  study  was  to  be  the  identifi- 
cation and  analysis  of  authoring  aids  usable  in  a military  training  environment, 
selected  documents  had  to  report  work  developed  in  a military  setting,  or  in  the 
civilian  sector  and  be  suitable  for  adaptation  to  operational  military  training. 
Third,  work  discussed  in  a report  had  to  be  applied  and  evaluated  or,  if  a procedure 
or  methodology,  at  least  applied  though  unevaluated.  Authoring  methodologies  were 
included  in  the  study  only  for  comparison  with  tools  and  procedures.  Only  one 
methodology  was  included  in  the  annotated  bibliography 

2.2.4  Procedure  for  Bibliographic  Compilation  - The  procedure  adopted  for 
compiling  the  annotated  bibliography  consisted  of  the  following  four  steps: 

2.2.4. 1 Definition  of  Item  Record  - An  item  record  was  defined  to  include  the 

characteristics  by  which  research  items  could  be  compared.  Data  elements  of  the 
item  record  included:  (a)  personal  or  corporate  author(s);  (b)  institutional 

affiliation;  (c)  publication  title;  (d)  publication  date;  (e)  media;  (f)  type  of 
application;  (g)  type  of  evaluation;  (h)  type  of  environment  (military  or  civilian); 
and  (i)  type  of  ISD  activity,  according  to  the  IPISD  model.  These  data  were  entered 
by  codes  which  allowed  quicker,  more  accurate  data  entry  and  reduced  item  record 
length.  For  example,  the  institution,  "University  of  Texas  at  Austin,"  was  coded 
"B2480." 

2. 2. 4. 2 Creation  of  Data  Base  - For  documents  read  initially  in  full-text 
form,  a complete  set  of  data  was  input  in  one  operation.  Data  entry  for  documents 
in  abstract  form  required  two  steps.  The  symbol  "ORD"  (which  indicated  that  an 
item  had  been  ordered)  in  the  institution  field  and  author  and  title  were  input 
to  establish  the  document's  item  record.  After  the  document  had  been  received,  it 
was  analyzed  to  obtain  the  remaining  data  required,  and  the  "ORD"  code  was  replaced 
by  the  institutional  affiliation  code.  Using  "ORD"  in  the  institution  field  auto- 
matically built  an  on-line  order  file  which  allowed  the  determination  of  the  number 
of  documents  on  order  versus  those  on  hand,  and  allowed  the  identification  of 
excessive  lags  between  orders  and  receipts  as  a basis  for  follow-up  action.  A 
special-purpose  program,  called  OLD  TABLE,  produced  a daily  total  of  item  records 
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in  each  ISD  activity  and  phase,  and  in  the  whole  data  base.  Systematic  use  of  this 
capability  made  it  possible  to  "map"  the  data  base  and  identify  those  ISD  activities 
or  phases  requiring  additional  research. 

2. 2. 4. 3 Analysis  - The  data  base  was  analyzed  in  terms  of  authoring  aids 

(tools,  procedures,  and  methodologies)  for  developing  instructional  materials. 
Printout  listings  were  obtained  of  research  reports  describing  authoring  aids  that 
had  been  applied  and  evaluated  in  military  or  civilian  environments.  In  some  cases, 
an  additional  printout  identifying  unevaluated  applications  was  also  produced. 
Appropriate  authoring  aids  were  then  selected  from  these  listings  based  on  three 
guidelines:  large  pools  of  applied  and  evaluated  reports  allowed  some  early  or 

obsolete  work  to  be  rejected;  relative  cost  and  availability  made  some  authoring 
aids  more  suitable  than  others;  and  approaches  and  concepts  that  had  been  widely 
used  but  not  evaluated  were  occasionally  chosen  over  narrower,  more  parochial  work 
that  had  been  evaluated. 

2. 2. 4. 4 Cataloguing  - The  selected  reports  were  categorized  by  the  following 
considerations: 

a.  Coded  as  a tool,  procedure,  or  methodology, 

b.  Coded  by  type  of  medium,  application,  evaluation,  environment,  and  ISD 
activity. 
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3.0  RESULTS 

Three  constraints  limit  the  interpretation  of  the  results.  First,  due  to 
limited  resources,  classification  and  categorization  of  the  reports  surveyed  were 
not  subject  to  inter-judge  reliability  tests.  Second,  the  population  of  reports 
in  the  data  base  was  created  by  applying  operational  definitions  of  authoring  tools, 
procedures,  and  methodologies,  and  the  classification  scheme  to  characterize  the 
reports  was  derived  by  the  author  of  this  study.  Third,  though  it  may  be  suggested 
that  documentation  patterns  for  the  total  ISD  R&D  effort  and  this  study  may  be 
similar  (particularly  for  that  portion  of  the  effort  applying  to  authoring  tools 
and  procedures),  no  specific  evidence  was  systematically  obtained  to  support  this. 

3.1  RESEARCH  HYPOTHESES 

To  test  the  four  hypotheses  posed  in  Section  2.1,  a total  of  1424  documents 
were  analyzed.  Two  pools  of  documents  relative  to  ISD  authoring  aids  were  collected: 
(a)  applied  and  evaluated;  and  (b)  applied  but  not  evaluated.  These  documents  were 
further  classified  by  environment,  either  military  or  civilian,  and  by  type  of 
authoring  aid.  The  numbers  of  documents  in  each  classification  are  presented  in 
Tables  1,  2,  and  3.  Chi  square  was  used  to  test  each  research  hypothesis. 

3.1.1  First  Hypothesis  - "The  civilian  environment  has  a greater  number  of 
applied,  unevaluated  documents  than  the  military  environment."  The  test  of  this 
hypothesis  was  statistically  significant  (X  = 172.13,  df  = 1,  p < .01),  indicating 
that  more  applied,  unevaluated  documents  were  associated  with  the  civilian 
environment.  (See  Table  1) 

3.1.2  Second  Hypothesis  - "The  civilian  environment  has  a greater  number  of 
applied,  evaluated  documents  than  the  military  environment."  The  test  of  this 
hypothesis  was  not  statistically  significant  (X^  = 1.73,  df  = 1,  p > .01).  This 
suggests  that  relatively  equal  numbers  of  applied,  evaluated  documents  exist  in 
each  environment.  (See  Table  1) 

3.1.3  Third  Hypothesis  - "There  are  unequal  numbers  of  documents  within  the 
three  authoring  aid  categories."  The  test  of  this  hypothesis  was  statistically 
significant  (X^  » 145.72,  df  = 2,  p < .01).  Based  on  the  raw  data,  it  is  apparent 
that  more  procedures  were  available  than  either  tools  or  methodologies.  (See  Table  2) 
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3.1.4  Fourth  Hypothesis  - "The  civilian  environment  has  a greater  number  of 
authoring  aids  than  the  military  environment."  The  test  of  this  hypothesis  only 
approached  statistical  significance  (X^  = 6.26,  df  = 1 , .05  > p > .01.  Inspection 
of  the  raw  data,  however,  suggests  a tendency  for  the  civilian  environment  to 
produce  a greater  number  of  authoring  aids.  (See  Table  3) 

3.2  AVAILABILITY  OF  AUTHORING  AIDS 

3.2.1  ISO  Activities  Lacking  An  Authoring  Aid  - The  raw  data  were  analyzed 

to  determine  the  ISO  activities  for  which  there  were  no  tools.  Tools  are  considered 
to  be  the  most  desirable  authoring  aid  because  they  are  readily  transportable  and 
require  the  least  adaptation  or  restructuring  by  potential  users.  No  tools  were 
found  for  the  following  ISO  activities: 

o Select  Tasks/Functions 

0 Construct  Job  Performance  Measures 

0 Select  Instructional  Settings 

0 Specify  Learning  Events/Activities 

0 Implement  Instructional  Management  Plan 

0 Conduct  External  Evaluation. 

No  authoring  procedures  were  found  for  the  ISO  activities: 

0 Select  Instructional  Setting 

0 Specify  Learning  Events/Activities. 

3.2.2  Availability  of  Authoring  Aids  by  ISP  Activity  - The  steps  of  the 
IPISO  model  are  specified  in  the  fold-out  in  Appendix  B.  Each  step  has  an  input 
and  an  output.  For  example,  one  phase  in  the  IPISD  model  is  "Analyze."  One 
step  in  that  phase  is  "Analyze  Job."  One  input  to  that  step  is  a needs  analysis 
imposed  by  Installation  of  a new  system,  weapon,  or  equipment  to  a job  or  training 
situation.  An  output  of  that  step  is  a validated  list  of  job  tasks  concerning 
that  new  component.  Although  the  output  of  one  step  may  provide  an  input  to  the 
following  step,  the  steps  of  the  IPISD  model  are  not  necessarily  linear  or  performed 
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ENVIRONMENT 

NUMBER 

Military 

189 

Civilian 

242 

TABLE  3:  NUMBERS  OF  AUTHORING  AIDS  BY  ENVIRONMENT 


CATEGORY 

NUMBER 

Tools 

67 

Procedures 

260 

104 

TABLE  2:  NUMBER  OF  AUTHORING  AIDS  BY  CATEGORY 


STATUS 

ENVIRONMENT 

IKRn^SS 

Civilian 

Unevaluated 

57 

308 

Evaluated 

297 

331 

Total 

354 

639 

TABLE  1 : NUMBER  OF  APPLIED  DOCUMENTS  SURVEYED 
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on  a once-and-once-only  basis.  The  static  model  only  provides  a useful  method  for 
working  on  or  talking  about  the  dynamic  process  called  ISD.  Since  the  authoring 
process  is  composed  of  four  activities  (information  gathering,  conversion, 
revision,  and  publication),  authoring  tools  and  procedures  were  drawn  from  ISD 
activities  which  analyzed,  designed,  developed,  implemented,  and  controlled  those 
activities.  Available  authoring  tools  and  procedures  that  follow  have  been  grouped 
according  to  ISD  activities.  Numbering  of  ISD  activities  is  in  accord  with  the 
fold-out  cited  above. 

1.1  Analyze  Job.  An  extensive  Air  Force  occupational  analysis  program  was 
begun  in  the  1960s  that  culminated  in  development  of  a very  effective  tool  for 
computer-based  job  analysis.  The  CODAP  system  (for  Comprehensive  Occupational 
Data  Analysis  Programs)  provides  individual  or  group  descriptions  that  specify 
the  percent  time  spent  on  each  task  in  a job,  describes  an  actual  job  at  different 
task  levels  such  as  apprentice  or  officer,  derives  individual  and  group  job  differ- 
ences, and  identifies  specific  types  of  jobs  as  they  exist  or  should  exist  within 

a general  specialty.  Detailed  descriptions  of  procedures,  programming  notes,  and 
applications  are  provided  by  Archer  (1966),  Morsch  and  Archer  (1967),  Christal 
(1972),  Weismuller,  Barton,  and  Rogers  (1974a,  1974b),  Stacey,  Weismuller,  Barton 
and  Rogers  (1974),  Mayo,  Nance,  and  Shiegekawa  (1974),  Stacey  and  Hazel  (1974), 
and  Christal  and  Weismuller  (1975). 

Foley  (1975)  describes  the  use  of  several  standardized  worksheets  for  manual 
job  analysis.  The  Task  Inventory  Exchange  (TIE,  1975)  collects  occupational  and 
task  inventories  and,  to  date,  has  published  three  task  inventory  directories. 

1.2  Select  Tasks/Functions.  McKnight  and  Adams  (1970)  report  that  a panel 

of  experts  selected  critical  behaviors  for  driver  education  training  from  a possible 
pool  of  1000  relevant  behaviors.  Powers  (1971)  added  a phrase  to  the  standard  task 
statement  which  aided  the  selection  of  tasks  for  teaching  physician's  assistants. 

The  phrase,  "in  order  to,"  was  used  to  indicate  an  intended  outcome  of  a task,  and, 
thereby,  group  students  according  to  the  required  levels  of  competence.  Cline  (1973) 
used  a multivariate  statistical  model  to  predetermine  preferable  aircraft  assign- 
ments for  Air  Force  pilots. 

1.3  Construct  Job  Performance  Measures.  Gael  (1974)  describes  the  selection 
and  design  of  instruments  used  to  measure  job  aptitudes  and  job  proficiencies  for 
five  Bell  System  jobs.  The  instruments  employed  both  knowledge  and  performance 
measures  as  well  as  simulation. 
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I. 4  Analyze  Existing  Course.  Lewis,  Willow,  Brock,  Lonigro,  Eschenbrenner, 

and  Hanson  (1974)  analyzed  an  Air  Force  course  in  Precision  Measuring  Equipment 
(PME)  in  order  to  convert  it  from  conventional  instruction  to  self-paced,  individu- 
alized instruction.  The  analysis  was  done  in  four  phases:  (a)  select  course 

segments  for  individualization;  (b)  analyze  course  and  job  documentation; 

(c)  choose  the  target  population;  and  (d)  examine  the  training  environment. 

Formal  analysis  of  an  existing  Air  Force  course  appears  in  the  Specialty  Training 
Standard  (STS),  Plan  of  Instruction  (POI),  and  the  Course  Chart  for  Inventory  Manage- 
ment Specialist  (645X0)  and  Materiel  Facilities  Specialist  (647X0).  The  documents, 
which  provide  codes  for  various  levels  of  performance  required  at  end  of  training 
and  divide  instruction  into  sequential  segments,  can  be  used  to  distinguish  between 
required  levels  of  performance  (both  cognitive  and  psychomotor)  and  to  sequence 
instruction  accordingly. 

II. l  Develop  Objectives.  A "grammed  workbook  intended  as  an  introduction 
to  behavioral  objectives  and  to  a Navy  handbook  which  teaches  the  writing  of 
behavioral  objectives  is  divided  into  three  modules  (Bureau  of  Naval  Personnel, 

1968).  Module  1 discusses  the  general  concept  of  behavioral  objectives.  Module  2 
introduces  the  Navy's  Handbook  for  Writing  Learning  Objectives,  and  Module  3 
contains  an  exercise  kit  for  using  the  handbook  and  includes  testing  instruments. 

The  Behavioral  Objectives  Training  Package  (ESCOE,  1971)  serves  as  a user's  guide 
to  writing  and  implementing  behavioral  objectives  and  provides  an  overview  of 
the  operation  of  the  Educational  Service  Center  for  Occupational  Education  which 
contains  a clearinghouse  for  behavioral  objectives  information.  Oillman  (1971) 
provides  an  excellent  discussion  of  writing  effective  behavioral  objectives.  The 
discussion  is  presented  in  programmed-text  format  with  self-reviews  at  the  end  of 
each  section.  Another  programmed  text  by  Rose,  Balasek,  Kelleher,  Lutz  and 
Nelken  (1972a,  1972b)  is  designed  to  help  teachers  learn  how  to  write  performance 
objectives.  An  instructor's  manual  accompanying  the  programmed  text  provides 
management  procedures  for  inservice  training.  Barton  (1973)  presents  a booklet 
that  teaches  how  to  recognize  and  write  technically  correct  performance  objectives. 
Validated  over  a two-year  period,  the  booklet  is  in  programmed-text  format  and 
contains  numerous  self-checks.  A compact,  reuseable,  multi-color  Reference 

Wall  Chart  (INS6R0UP,  1975)  identifies  the  structure  of  measurable  objectives, 
discusses  three  applications  of  measurable  objectives,  presents  several  taxonomies 
of  learning  which  are  used  to  analyze  objectives,  and  contains  a brief  section  of 
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annotations  for  the  taxonomies.  Also  on  the  chart  are  three  addresses  where 
collections  of  objectives  may  be  obtained. 

11. 2 Develop  Tests.  Brown,  Brooks,  Cocks,  and  Kersch  (1972)  report  a 
teacher  workshop  for  defining,  developing,  and  evaluating  test  Items.  The  Center 
for  the  Study  of  Evaluation,  In  conjunction  with  Research  for  Better  Schools  Inc., 
analyzed  and  evaluated  approximately  2600  tests  for  measuring  higher-order  cognitive, 
affective,  and  interpersonal  skills  (Center  for  the  Study  of  Evaluation,  1972). 

A practical  guide  for  preparing  and  using  criterion-referenced  tests  in  classroom 
instruction  was  developed  by  Gronlund  (1973).  Checklists  for  evaluating  criterion- 
referenced  tests  and  procedures  for  item  analysis  are  also  given.  Swezey  and 
Pearl  stein  (1974)  developed  a manual  that  thoroughly  covers  the  design,  use  and 
evaluation  of  criterion-referenced  tests.  Four  steps  are  discussed:  (a)  develop 

adequate  behavioral  objectives;  (b)  develop  a test  plan  and  examine  practical 
constraints;  (c)  build  an  item  pool;  and  (d)  select  the  final  test  items  for 
instruction.  Stevens  and  O'Neil  (1976a)  discuss  procedures  for  generating  test 
items  by  item  transformation,  i.e.,  by  varying  sentences  according  to  their 
syntactic  structure.  The  test  items  can  be  used  in  multiple-choice  or  constructed 
response  tests.  Stevens  and  O'Neil  (1976b,  1976c)  suggest  procedures  for  develop- 
ing and  evaluating  tests  items  for  generalized  multiple-choice  tests  and  for 
comprehension-,  application-,  and  analysis-level  tests. 

1 1. 3 Describe  Entry  Behavior.  Bierbaum  and  Planisek  (1969)  developed  an 
index  and  procedure  for  determining  the  probable  academic  success  of  students 
seeking  readmission  after  academic  dismissal.  An  interest  inventory  designed  to 
measure  the  vocational  interests  of  enlisted  men  entering  the  Air  Force  was 
developed  by  Ecternacht,  Reilly,  and  McCaffrey  (1973).  Called  VOICE  (for  Vocational 
and  Occupational  Interest  Choice  Examination),  the  instrument  was  to  be  used  in 

the  development  of  an  interest  inventory  for  the  Guaranteed  Enlistment  Program. 
Mockovak  (1974)  assessed  the  literacy  requirements  of  Air  Force  career  ladders 
in  order  to  determine  the  reading  demands  placed  upon  trainees  and  job  incumbents 
by  instructional  materials.  Vitola,  Mullins,  and  Brokaw  (1973)  found  that  removal 
of  the  draft  as  a motivator  for  enlistment  affected  the  Air  Force  in  three  ways: 

(a)  decreased  accessions  completing  education  beyond  the  high  school  level ; (b) 
the  discharge  rate  for  first-term  airmen  after  four  years  was  79  percent;  and  (c) 
the  average  difference  on  mean  scores  for  AQE  aptitude  composites  between  Blacks 
and  Non-blacks  decreased. 
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11*4  Determine  Sequence  and  Structure.  Dansereau,  Evans,  Wright,  Long,  and 
Actkinson  (1974)  and  Dansereau,  Evans,  Actkinson,  and  Long  (1974)  developed  and 
evaluated  the  use  of  multidimensional  scaling  (called  INSCAL)  as  a procedure  for 
describing  the  information  structure  of  Air  Force  instructional  materials  and 
generating  effective  instructional  sequences.  A checklist  for  determining 
students'  positions  along  a curriculum  scale  was  developed  (Bucks  County  Public 
Schools,  1970)  which  may  be  useful  as  a manual  backup  to  computer-based  learning 
systems  for  assessing  student  performance  and  determining  proper  learning 
sequences. 

1 1 1. 3 Review/Select  Existing  Materials.  The  State  University  College  at 
Buffalo  (1972)  has  published  a user's  manual  which  describes  computer-based  resource 
units  in  various  subject  areas  and  gives  directions  for  obtaining  them.  Up  to  five 
units  may  be  requested,  and  individual  difference  variables  may  be  indicated  on  the 
request  form.  An  annotated  listing  of  curriculum  materials  which  can  be  obtained 
from  state  education  agencies  is  available  from  the  U.S.  Government  Printing 
Office  (1973).  Various  guides  cover  agriculture,  distributive  education,  health 
occupations,  home  economics,  technical  education,  and  trade  and  industrial  occupa- 
tions. The  National  Network  for  Curriculum  Coordination  in  Vocational-Technical 
Education  develops,  publishes,  and  distributes  curriculum  guides  and  various 
curriculum  support  services  from  six  regional  curriculum  management  centers. 
Curriculum  guides  are  published  in  the  form  of  brochures,  booklets,  and  microfiche. 
The  Special  Reserve  Components  Educational  Video  Tape  Catalog  contains  a listing 
of  educational  video  tape  programs  selected  for  Army  unit  training.  Though  the 
materials  are  intended  for  Reserve  component  use,  they  may  be  used  by  other  Services, 
Army  schools,  and  training  centers.  The  video  tapes  may  be  borrowed  from  various 
Training  Aids  Support  Offices  (TASOs)  or  can  be  copied  on  user-supplied  cassette 
tapes.  The  Directorate  for  Audio-Visual  Activities  (DAVA,  1975)  presents  a detailed 
user's  guide  that  provides  the  required  information  for  participating  in  the 
Defense  Audio-Visual  Information  Network  and  for  utilizing  the  network's  Audio- 
Visual  Products  Data  Base.  Appendices  give  input  data  guidelines  for 
data  coding  and  other  computer-related  activities.  The  Department  of  the  Army 
(1974,  1975)  provides  two  catalogs  of  Army  motion  picture  films  and  audio-visual 
aids.  The  first  catalog  gives  annotated  listings  of  motion  picture  films,  slides, 
audio-visual  kits,  recordings,  and  tape  recordings  intended  for  use  by  instructors, 
information  officers,  and  defense  contractors.  The  second  catalog  provides  annota- 
tions of  motion  picture  films  for  non-profit  use  by  Government  agencies,  civic, 
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religious,  fraternal  and  educational  organizations,  schools,  colleges,  and  uni- 
versities. Materials  from  the  first  catalog  are  oriented  to  Army  career  fields, 
while  those  in  the  second  catalog  are  on  more  general  subjects  such  as  education, 
sports,  and  health, 

111. 4 Develop  Instruction.  Best  (1971)  discusses  various  procedures  and 
materials  which  minimize  expense  and  time  in  the  preparation  and  production  of 
computer-controlled  films.  For  local  production  shops,  this  report  provides  much 
of  the  information  required  for  processing  filmstrips  into  slides  for  either 
computer-controlled,  rear-screen  projectors  or  for  transparencies.  Thiagarajan 
(1971)  offers  a programmed  text  for  teaching  the  craft  of  programming.  Five 
stages  of  the  programming  process  are  discussed,  and  a confirmation  section  at 
the  conclusion  of  the  book  contains  correct  answers  to  the  questions  embedded  in 
the  text.  How  to  Design  and  Produce  Individualized  Instruction  Programs 
(Minnesota  Mining  and  Manufacturing  Co.,  1972)  is  a pragmatic  workbook 
intended  to  be  used  for  developing  audio-visual  programs  for  the  3M  Sound-On- 
Slide  System.  In  another  user's  manual,  guidelines  for  writing  computer-based 
resource  units  are  presented  and  organized  into  two  operations  (State  University 

of  New  York,  1972).  The  first  operation  involves  formulating  behavioral  objectives, 
instructional  content,  activities,  materials,  and  measuring  devices.  The  second 
operation  involves  coding  these  five  component  items  for  future  computer  retrieval. 
Deterline  and  Lenn  (1972)  developed  a self-study  course  that  teaches  a set  of 
skills  called  CISTRAIN  (for  Coordinated  Instructional  ^stem  for  Training)  for 
designing,  developing,  and  implementing  instruction.  The  two-volume  set  of 
instruction  includes  a Lesson  Book  and  a Study  Resource  Materials  Book.  Siegel, 
Lambert,  and  Burkett  (1974)  present  a concise,  entertaining  synthesis  of  readability 
research  and  illustrate  findings  from  that  field  that  are  useful  to  course  writers. 
Nesbit  and  O'Neil  (1976)  provide  guidelines  for  editing  programmed  texts.  The 
report  contains  a glossary  of  terms,  explanations  of  three  models  for  programmed 
texts,  and  guidelines  for  checking  the  lesson  content  and  author's  style. 

111. 5 Validate  Instruction.  The  use  of  computer-based  test  analyses  and  course 
evaluation  questionnaires  is  reported  by  Aleamoni  (1970).  The  system  provides  for: 
(a)  test  scores;  (b)  analysis  of  scores  for  groups  or  individuals;  (c)  test 

item  analysis  printouts;  (d)  item  analysis  data;  and  (e)  analysis  of  course 
evaluation  questionnaires,  attitude  scales,  and  other  measures  which  have  no 
known  correct  response.  A manual  and  system  documentation  are  provided.  Abedor 
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(1972)  presents  a formative  evaluation  model  which  functions  in  a real-world, 
operational  environment.  Noting  that  multiple,  iterative  revisions  of  materials 
can  be  a monumental  and  costly  effort,  Abedor  designed  a formative  evaluation 
model  to  generate  large  amounts  of  data  on  instructional  problems  in  a one-shot 
trial  of  the  prototype  lesson.  Ward  (1972)  investigated  an  author's  use  of  forma- 
tive evaluation  during  the  preparation  of  a CAl  course.  Rayner  (1972)  developed 
a formative  evaluation  model  that  provides  data  not  only  for  content  but  also  for 
procedural  changes  to  instructional  materials.  Hummel-Rossi  (1972)  devised  a 
multiple-choice  test  for  evaluating  knowledge  of  Bloom's  taxonomy.  The  test 
includes  an  instruction  sheet,  a specially- treated  answer  sheet  which,  when 
rubbed,  displays  the  answer,  and  the  testing  instrument. 

_V.l  Conduct  Internal  Evaluation.  Bailey  (1972)  discusses  the  testing  of 
manual  procedures  in  the  Bell  System  personnel  subsystem.  The  procedures  consist 
of  six  steps  and  provide  data  concerning  the  numbers  and  kinds  of  errors  and 
the  probable  causes  of  errors  for  various  job  activities.  Miller  and  Sellman 
(1973)  developed  a new  student  critique  form  for  the  Air  Training  Command  which 
indicated  that  (a)  attitudes  of  trainees  toward  technical  training  were  "slightly 
positive;"  (b)  the  most  positive  attitudes  toward  technical  training  were  held 
by  Non-Commissioned  Officers,  followed  by  Officers,  and  then  Airmen;  and  (c) 
airmen  were  uncertain  that  technical  training  was  adequately  preparing  them  for 
on-the-job  performance. 

V.2  Conduct  External  Evaluation.  Anastasio  (1972)  proposed  a field-test 
evaluation  of  the  PLATO  and  TICCIT  systems  that  included  cost  and  educational 
analyses.  Of  interest  to  course  writers  are  the  discussions  of  achievement  and 
attitude  measures  and  the  Survey  of  Instructor  Activities  and  Attitudes  contained 
in  Appendix  A of  the  Anastasio  report. 
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The  preceding  findings  suggest  four  areas  for  further  consideration.  First, 
an  ISO  activity  such  as  Select  Instructional  Setting  that  has  no  tool  or  procedure 
associated  with  it  should  be  investigated  to  determine  whet■^•*r  a tool  or  procedure 
is  critical  to  the  role  of  that  ISO  activity  in  education  or  training.  If  a tool 
or  procedure  is  critical  to  the  execution  of  that  activity,  further  investigation 
should  be  conducted  in  order  to  create  the  tool  or  procedure. 

Second,  if  a tool  is  generally  considered  the  most  useful  authoring  aid  for 
an  ISO  activity,  then  the  proportion  of  procedures  identified  in  the  present  study 
should  be  modified  into  tools. 

Third,  in  an  ISO  activity  where  several  tools  and  procedures  but  few  method- 
ologies are  available,  priority  should  be  given  to  further  research  and  development. 
In  test  development,  for  example,  nineteen  tools  and  procedures  were  identified  for 
constructing  criterion-referenced  tests  which  are  currently  the  most  widely-used 
form  of  testing  instrument.  However,  test  development  is  identified  with  only  six 
methodologies.  When  new  situations  arise  where  criterion-referenced  testing  is 
not  appropriate  and  few  test  development  methodologies  exist,  an  unacceptable  lag 
between  formulation  of  new  testing  systems  may  occur. 

Fourth,  during  the  course  of  the  study,  six  significant  aspects  of  the  author- 
ing process  were  identified:  (a)  size  and  configuration  of  authoring  teams;  (b) 

knowledge  and  proficiency  of  course  authors;  (c)  existing  course  structures 
including  lesson  materials  and  course  control  documents;  (d)  type  of  media  and 
types  of  instruction;  (e)  extent  and  type  of  evaluation  and  revision;  and  (f)  text- 
handling and  reproduction  systems.  The  only  subjective  aspects  of  the  authoring 
process  are  knowledge  and  proficiency  of  course  authors.  Since  identification 
and  distribution  of  authoring  tools  and  procedures  would  allow  course  authors  to 
develop  materials  from  a baseline  of  equal  knowledge,  the  objective  aspects  of  the 
author  process  can  be  categorized  and  manipulated  in  a "laboratory"  (i.e.,  con- 
trolled) setting.  Thus,  it  should  be  possible  to  model  authoring  process  variables 
in  order  to  formulate  decision  rules  for  managing  the  materials  development  cycle 
with  an  optimum  mix  of  personnel  and  organization. 
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In  summary,  ISO  has  been  discussed  as  an  adaptation  of  the  systems  approach. 
The  product  of  an  ISO  effort  was  seen  as  an  instructional  system,  one  component 
of  which  was  the  authoring  process.  Four  activities  of  the  authoring  process 
were  identified:  information  gathering,  conversion,  revision,  and  publication. 

Authoring  tools  and  procedures  were  identified  for  each  of  the  four  activities. 

It  was  recommended  that  an  authoring  process  model  be  developed  for  formulating 
decision  rules  for  properly  managing  development  of  instructional  materials. 
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Data  Item  Names  and  Codes  for  Document  Entries 
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DATA  ITEM  NAME 


DATA  ITEM  CODE 


Author 

Institution 


Title 

Date  of  Publication 
ISD  Activity  Number 
Type  of  Media 


Type  of  Application 


Type  of  Evaluation 
Type  of  Environment 


Self-explanatory 
B = College  or  University® 

Y = Non-profit  Organization 
Z = Profit-making  Organization 
Self-Explanatory 
Self-Explanatory 

2.2b 

0 = ETV 

1 = Radio 

2 = Programmed  Text 

3 = CAI 

4 = CMI 

5 = CAI/CMI 

6 = Audio-visual 

7 = Lecture 

t 

8 = Other 

0 = Theory 

1 = Applied 

2 = Applied  in  Lockstep 

3 = Applied  in  Self-paced 

0 = Evaluated 

1 = Not  evaluated 

0 = Civilian 

1 = Joint  Military  Service 

2 = Army 

3 = Navy 

4 = Air  Force 

5 = Marines 

6 = None 
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, APPENDIX  A 
(Continued) 


^College  and  university  codes  were  followed  by  a four-digit  numeric 
code  obtained  from  the  Skills  Inventory  System  Code  Book  (St.  Louis: 
McDonnell  Douglas  Corp. , 1974).  Profit  and  non-profit-making  organiza- 
tion codes  were  followed  by  that  organization's  accepted  abbreviation. 
For  example,  ZIBM  indicated  the  International  Business  Machines  Corpora- 
tion and  YYMCA  indicated  the  Young  Men's  Christian  Association.  Where 
no  standard  abbreviation  could  be  obtained,  only  the  one-letter  code 
was  used. 

\ 

^ISD  activity  numbers  follow  those  of  the  Interservice  Procedures  for 
Instructional  Systems  Development  Model  shown  in  Figure  1.  For  example, 
2.2  indicates  "Develop  Tests." 
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APPENDIX  B 

AUTHORING  AID  ANNOTATED  BIBLIOGRAPHY 

The  abstracts  in  the  bibliography  are  grouped  in  accordance  with  the  steps  of 
the  IPISD  model.  The  IPISD  model  consists  of  five  phases  and  nineteen  steps;  the 
phases  are  Analysis,  Design,  Development,  Implementation,  and  Control.  The  steps 
within  each  phase  are  summarized  below. 

Phase  I.  Analyze.  A certain  training  or  educational  program  is  determined 
to  be  both  desirable  and  feasible.  The  job  or  preferred  outcome  to  be  taught  is 
analyzed  as  a series  of  task  statements.  Numerical,  analytic  techniques  and  the 
judgment  of  subject  matter  experts  are  used  to  select  a subset  of  the  task  state- 
ments for  training  and  instruction.  The  instructional  setting,  i.e.,  whether  the 
training  and  instruction  should  be  conducted  in  formal  schools,  on  job  situations, 
in  correspondence  courses,  or  wherever,  is  determined. 

Phase  II.  Design.  The  tasks  selected  for  training  and  instruction  in  Phase 
I are  converted  into  behavioral  objectives  which  specify  the  outcomes  to  be  obtained 
at  the  end  of  training.  Criterion  measures  are  devised  for  measuring  learner  per- 
formance (both  cognitive  and  psychomotor,  if  desired)  against  the  objectives. 
Incoming  student  characteristics  are  identified  in  terms  of  general  ability,  reading 
level,  and  prior  experience. 

Phase  III.  Development.  Carefully  selected  and  prepared  media  and  materials 
are  developed.  Decisions  are  made  concerning  (a)  management  of  students,  instruc- 
tors, and  course;  (b)  desired  learning  experiences  and  activities;  and  (c)  the 
form  and  content  of  the  instructional  delivery  system.  Instructional  materials 
are  developed  and  validated  during  small-  and  large-group  tryouts.  Instructional 
materials  and  procedures  are  revised  in  order  to  assure  that  learners  meet  lesson 
and  course  objectives. 

Phase  IV.  Implementation.  Instructional  and  management  staff  are  trained 
to  meet  the  requirements  of  the  instructional  management  plan  and  delivery  system 
selected  as  the  instructional  vehicle.  Media  and  materials  are  implemented  in 
approved,  large-scale  courses  of  instruction,  and  feedback  is  obtained  on  both 

system  performance  and  effectiveness  of  materials. 
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Phase  V.  Control . Data  obtained  during  instruction  are  used  to  ansv#er  tv«) 
questions.  First,  are  students  learning  what  they  were  intended  to  learn;  and 
second,  did  what  the  students  learn  benefit  them  after  training,  i.e.,  during 
later  employment  or  further  study?  Negative  answers  to  either  question,  which 
are  identified  during  internal  and  external  (or  field)  evaluations,  indicate  that 
the  contents  and  procedures  of  the  new  system  should  be  revised. 

Each  abstract  in  the  bibliography  is  prefaced  by  a three-part  number  and  a 
step  title.  The  reference  number  on  the  left  side  of  the  page  indicates  the  ISO 
phase,  the  step  in  that  phase,  and  the  particular  abstract  in  that  step.  The  step 
title  on  the  right  side  of  the  page  provides  an  easy  reference  point  for  the  reader. 
For  example,  an  abstract  prefaced  with  the  number  "3.1.2"  indicates  that  the 
abstract  is  concerned  with  Phase  3 (Development),  Step  1 (Developing  Objectives) 
and  that  it  is  the  second  abstract  in  that  step: 


The  step  title  translates  the  code  "3.1"  into  "Specify  Learning  Events/Activities." 
To  retrieve  information  concerning  a particular  ISD  activity  or  step,  merely  con- 
sult the  Table  of  Contents  of  the  bibliography  and  turn  to  the  section  of  abstracts 
beginning  with  the  appropriate  Phase/Step  number.  The  abstracts  that  follow 
begin  with  Phase  1,  Step  1,  Abstract  1 (1.1.1)  and  continue  through  with  Phase  5, 
Step  3,  Abstract  2 (5.3.2).  Refer  to  the  fold-out  on  the  following  page  as  neces- 
sary for  identification  and  sequence  of  ISD  activities  and  steps. 
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PHASE  1 

ANALYZE 


PHASE  2 

DESIGN 


PHASE  3 

DEVELOP 


PHASE  4 

IMPLEMENT 


PHASE  5 

CONTROL 


THE  FIVE  PHASES  ARE: 


( 

Inputs,  processes,  and  outputs  in  Pho»«  I are  all  based  on  job  information.  An  inventory  of  job 
tasks  is  compiled  and  divided  into  two  groups:  tasks  not  selected  for  instruction  and  tasks 
selected  for  instruction  Performance  standards  for  tasks  selected  for  instruction  are  determined 
by  interview  or  observation  at  job  sites  and  verified  by  subject  matter  experts.  The  analysis  of 
existing  course  documentation  is  dorse  to  determine  if  all  or  portions  of  the  analysis  phase  and 
other  phases  have  already  been  done  by  someone  else  following  the  ISO  guidelines.  As  a final 
analysis  phase  step,  the  list  of  tasks  selected  for  instruction  is  analyted  for  the  most  suitable 
instructional  setting  for  each  task. 


Beginning  with  Phot*  2.  the  ISO  model  is  corKerned  with  designing  instruction  using  the  job 
analysis  information  from  Phox  I.  The  first  step  is  the  conversion  of  each  task  selected  for 
training  into  a terminal  learning  objective  Each  terminal  learning  objective  is  then  analyzed  to 
determine  learning  objectives  and  learning  steps  necessary  for  mastery  of  the  terminal  learnirrg 
objective.  Tests  are  designed,  to  match  the  learning  objectives.  A sample  of  students  is  tested  to 
insure  that  their  entry  behaviors  match  the  level  of  learning  analysis.  Finally,  a sequerKe  of 
instruction  is  designed  for  the  learning  ob|ectives. 


The  instructional  development  phase  begins  with  the  classification  of  learning  objectives  by 
learning  category  so  as  to  identify  learning  guidelines  necessary  for  optimum  learning  to  take 
place.  Determining  how  instruction  is  to  be  packaged  and  presented  to  the  student  is 
accomplished  through  a media  selection  process  which  takes  into  account  such  factors  as 
learning  category  and  guideline,  media  characteristics,  training  setting  criteria,  and  costs. 
Instructional  management  plans  ate  developed  to  allocate  and  manage  all  resources  for 
conducting  instruction  Instructional  materials  are  selected  or  developed  and  tried  out.  When 
materials  have  been  validated  on  the  basis  of  empirical  data  obtained  from  groups  of  typical 
students,  the  course  is  ready  for  implententation 


Staff  training  is  required  for  the  implementation  of  the  instructional  management  plan  and  the 
instruction  Some  key  personnel  must  be  trained  to  be  managers  in  the  specified  management 
plan.  The  instructional  staff  must  be  trained  to  coixfuct  the  instruction  arxi  collect  evaluative 
data  on  all  of  the  instructional  corrtponents.  At  the  completion  of  each  instructional  cycle, 
management  staff  should  be  able  to  use  the  collected  information  to  improve  the  instructional 
system. 


Evaluation  and  revision  of  instruction  are  carried  out  by  personnel  who  preferably  are  neither 
the  instructional  designers  not  the  managers  of  the  course  under  study.  The  first  activity 
(internal  evaluation)  is  the  analysis  of  learner  performance  in  the  course  to  determine  instances 
of  deficient  or  irrelevant  instruction  The  evaluation  team  then  suggests  solutions  for  the 
problems  In  the  external  evaluation,  personnel  assess  job  task  performance  on  the  job  to 
determine  the  actual  performance  of  course  graduates  and  other  job  incumbents  All  collected 
data,  internal  and  axtarnal.csn  be  used  as  quality  control  on  instruction  and  as  input  to  any 
phase  of  the  system  for  revision 
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THE  BLOCKS  IN  EACH  PHASE  ARE: 


THE  OUTC< 


u 

» ANALYZE 
JOB 


3.1 

SPEOEY 

^EARNING 

EVENTS/ 

ACTIVITIES 


1.2 

^SELECT 
“ TASKS/ 
FUNCTIONS 


1.3 

CONSTRUCT 

^JOB 

PERFORMANCE 

MEASURES 


1.4 

.ANALYZE 

^EXISTING 

COURSES 


1.5 

■ SELECT 

^INSTRUCTIONAL 

SETTING 


2.3 

^DEVELOP 

^DEVELOP 

^DESCRIBE 

Gentry 

BEHAVIOR 

~ OBJECTIVES 

^ TESTS 

2.4 

DETERMINE 
SEQUENCE  & 
STRUCTURE 


32 

SPECIFY 
INSTRUCTION 
I MANAGEMENT 
PLANA 
DELIVERY 
SYSTEM 


Al 

IMPLEMENT 

^instructional 

MANAGEMENT 

PLAN 


4.2 

^CONDUCT 

INSTRUCTION 


31 

■CONDUCT 

INTERNAL 

EVALUATION 


32 

■ CONDUCT 
^EXTERNAL 
EVALUATION 


33 

.REVIEW/SELECT 

'existing 

MATERIALS 


34 

I DEVELOP 
INSTRUCTION 


3.5 

F VALIDATE 
INSTRUCTION 


33 

F REVISE 
SYSTEM 


a list  of  tasks  pe< 
a list  of  tasks  seii 
a job  performan< 
an  analysis  of  ti 
construction  for 
usable  in  whole  r 

selection  of  the  r 


the  classification  | 
identification  of  ai 


.2 . . . the  media  seleclii] 
management  plan  I 

.3.  . . the  analysis  of  p/ 
learning  objectives 

.4.  . . the  development  * 
materials  are  not  a< 

.5 . . . field  tested  and  rev 


.1  . . . documents  contair 
resources,  and  staf 

.2  . . a completed  cycle 
the  succeeding  eye 


.1 . . . data  on  instruction 
.2 . . data  on  job  perforr 
.3.  . . instructional  syste 


3 


THE  OUTCOMES  OP  THE  BLOCKS  are: 


1 


,1  a list  of  tasks  performed  in  a particular  job. 

.2  a list  of  tasks  selected  for  training. 

.3  » |Ob  performance  measure  for  each  task  selected  for  instruction. 

.4.  an  analysis  of  the  job  analysis,  task  selection,  and  performance  measure 
construction  for  any  existing  instruction  to  determine  if  these  courses  are 
usable  in  whole  or  in  part. 

.5.  . selection  of  the  Instructional  setting  for  task  selected  for  instruction. 


2 


.1  . . . a learning  objective  for  and  a learning  analysis  of  each  task  selected  for 
instruction. 

.2  . . . test  items  to  measure  each  learning  objective. 

.3  . . . a test  of  entry  behaviors  to  see  if  the  original  assumptions  were  correct. 

.4  . . . the  sequencing  of  all  dependent  tasks. 


3 


,1  . . . the  classification  of  learning  objectives  by  learning  category  and  the 
identification  of  appropriate  learning  guidelines. 

.2 . . . the  media  selections  for  instructional  development  and  the  instructional 
management  plan  for  conductirtg  the  instruction. 

.3.  . . the  analysis  of  packages  of  any  existing  instruction  that  meets  the  given 
learning  objectives. 

.4.  . the  development  of  instruction  for  all  learning  objectives  where  existing 
materials  are  not  available. 

.5.  . . field  tested  and  revised  instructional  materials. 


4 


.1 . . . documents  containing  information  on  time,  space,  student  and  instructional 
resources,  and  staff  trained  to  conduct  the  instruction. 

.2 . . . a completed  cycle  of  instruction  with  information  needed  to  improve  it  for 
the  succeeding  cycle. 


5 


,1 . . . data  on  instructional  effectiveness. 

,2  . . tfata  on  job  performance  in  the  field. 

.3. . . instructional  system  revised  on  basis  of  empirical  data. 
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REFERENCE  NUMBER:  1.1.1 

ANALYZE  JOB 

CITATIONS: 

Archer,  W.  B.  Computation  of  group  job  descriptions  from  occupational  survey  data 
(Report  No.  PRL-TR-66-12).  Lackland  Air  Force  Base,  Texas:  Air  Force  Systems 
Command,  Aerospace  Medical  Division,  Personnel  Research  Laboratory,  December 
1966. 

Morsch,  J.  E.,  & Archer,  W.  B.  Procedural  guide  for  conducting  occupational  surveys 
in  the  United  States  Air  Force  (Report  No.  PRL-67-11).  Lackland  Air  Force  Base, 
Texas:  Air  Force  Systems  Command,  Aerospace  Medical  Division,  Personnel 
Research  Laboratory,  September  1967. 

Christal,  R.  E.  CODAP:  Input  standard  (INPSTD)  and  variable  generation  (VARGEN) 

programs  (Report  No.  AFHRL-TR-72-51 ).  Lackland  Air  Force  Base,  Texas:  Air 
Force  Human  Resources  Laboratory,  Personnel  Research  Division,  May  1972. 

Stacey,  W.  D.,  Weissmuller,  J.  J.,  Barton,  B.  B.,  4 Rogers,  C.  R.  CODAP:  Control 
card  specifications  for  the  UNIVAC  1108  (Report  No.  AFHRL-TR-74-84).  Lackland 
Air  Force  Base,  Texas:  Air  Force  Human  Resources  Laboratory,  Computational 
Sciences  Division,  October  1974. 

Weissmuller,  J.  J.,  Barton,  B.  B.,  4 Rogers,  C.  R.  CODAP:  Source  program  listings 

for  the  UNIVAC  1108  (Report  No.  AFHRL-TR-74-83).  Lackland  Air  Force  Base, 

Texas:  Air  Force  Human  Resources  Laboratory,  Computational  Sciences  Division, 
October  1974a. 

Weissmuller,  J.  J.,  Barton,  B.  B.,  4 Rogers,  C.  R.  CODAP:  Programmer  notes  for 

the  subroutine  library  on  the  UNIVAC  1108  (Report  No.  AFHRL-TR-74-85). 

Lackland  Air  Force  Base,  Texas:  Air  Force  Human  Resources  Laboratory,  Computa- 
tional Sciences  Division,  October  1974b. 

Mayo,  C.  C.,  Nance,  D.  M.,  4 Shigekawa,  L.  Evaluation  of  the  job  inventory 

approach  in  analyzing  USAF  officer  utilization  fields  (Report  No.  AFHRL-TR-75- 
22).  Lackland  Air  Force  Base,  Texas:  Air  Force  Human  Resources  Laboratory, 
Occupational  and  Manpower  Research  Division,  June  1975. 
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Stacy,  W.  J.,  & Hazel,  J.  T.  A method  of  determining  desirable  task  experiences  for 
first- line  supervisors  (Report  No.  AFHRL-TR-75-23).  Lackland  Air  Force  Base, 
Texas:  Air  Force  Human  Resources  Laboratory,  Occupational  and  Manpower 
Research  Division,  August  1975. 

Christal,  E.  E.,  & Weissmuller,  J.J.  New  CODAP  programs  for  analyzing  task  factor 
information.  Paper  presented  at  the  meeting  of  the  Military  Testing  Association, 
Indianapolis,  September  1975. 

ABSTRACT: 

An  extensive  Air  Force  occupational  analysis  program  was  begun  in  the  1960s  that 
culminated  in  development  of  an  on-going  computer-based  occupational  analysis  sys- 
tem. Providing  valid  job  analysis  information,  the  Comprehensive  O^ccupational  Data 
Analysis  P^rograms  (CODAP)  allow  job  analysts  to  describe  and  evaluate  existing  jobs 
within  Air  Force  Career  Specialties,  and  to  establish  training  priorities  for  new 
courses.  The  CODAP  system  can:  (a)  provide  individual  or  group  descriptions  that 
specify  the  percent  time  spent  on  each  task  in  a job;  (b)  describe  an  actual  job  at 
different  task  levels  (such  as  apprentice,  officer,  and  so  forth);  (c)  derive  indi- 
vidual and  group  job  differences;  and  (d)  identify  specific  types  of  jobs  as  they 
exist  or  should  exist  within  a general  specialty.  These  features,  as  well  as  an 
introduction  and  program  notes,  are  described  in  the  series  of  citations  discussed 
below. 

Archer  (1966)  provides  a detailed  analysis  of  occupational  survey  data  by  using 
a miniature  sample.  Based  on  responses  of  ten  incumbents  to  a job  inventory  con- 
sisting of  ten  task  statements,  composite  job  descriptions  are  derived  for  (a) 
special  groups  of  incumbents,  selected  on  the  basis  of  background  information  data; 
and  (b)  job  type  members,  identified  by  an  automated  job  clustering  program.  Illus- 
trations of  computer  outputs  from  both  types  of  analysis  are  explained.  The  10 
incumbents  are  Medical  Service  personnel  with  duty  level  ratings  of  three,  five,  and 
seven.  The  ten  task  statements  are  concerned  with  supervising  subordinates,  perform- 
ing ward  procedures,  and  performing  outpatient  procedures.  Each  incumbent  filled 
out  a job  inventory  survey  by  checking  the  tasks  he  performed  as  part  of  his  job 
and  rating  the  tasks  on  a nine-point  relative  time-spent  scale.  In  a very  detailed 
discussion  with  38  tables  and  four  figures,  the  author  illustrates  how  task-and- 
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duty-level  descriptions  are  derived,  and  how  membership  and  homogeneity  of  job  types 
are  clustered. 

Morsch  and  Archer  (1967)  describe  the  procedures  for  collecting,  organizing, 
analyzing,  and  reporting  job  task  information  of  Air  Force  officers  and  airmen. 
Discrete  steps  in  the  application  of  the  Air  Force  job  analysis  method  are  given  in 
the  following  chronological  order:  (a)  location  and  procurement  of  source  materials; 
(b)  construction  of  preliminary  job  inventory;  (c)  internal  review  of  preliminary 
job  inventory,  (d)  revision  of  preliminary  job  inventory;  (e)  field  test/review  of 
revised  job  inventory;  (f)  construction  of  final  job  inventory;  (g)  selection  and 
location  of  occupational  survey  samples;  (h)  reproduction  of  job  inventory  materials; 
(i)  packaging  and  mailing  job  inventories;  (j)  administration  of  job  inventories; 

(k)  receiving,  scanning,  and  coding  job  inventory  booklets;  (1)  integrating  added 
job  inventory  information;  (m)  keypunching  and  verifying  job  inventory  data;  (n) 
computer  analysis  of  job  inventory  data;  and  (o)  the  Occupational  Survey  Report. 

With  clear,  concise  discussion  of  each  step  plus  fifteen  appendices  which  contain 
everything  from  instructions  for  self-  or  group-administration  of  the  job  inventory 
to  a sample  letter  of  appreciation,  this  report  is  an  evaluated,  self-contained  tool 
for  job  or  task  analyzers. 

The  next  four  reports  present  information  to  be  used  by  computer  specialists 
aiding  the  job  analysis  effort.  Christal  (1972)  describes  and  gives  application 
examples  of  COOAP's  Input  Standard  Program  (INPSTD)  and  Variable  Generation  Program 
(VARGEN).  Stacey,  Weissmuller,  Barton,  and  Rogers  (1974)  present  control  card 
write-ups  for  34  main  programs  in  the  UNIVAC  1108  version  of  CODAP.  Arranged 
alphabetically  by  program  name  and  preceded  by  a short  summary  of  the  program 
functions,  each  write-up  contains:  (a)  detailed  control  card  specifications  neces- 
sary for  using  the  CODAP  system;  (b)  general  input-output  specifications;  (c)  general 
program  logic;  and  (d)  diagnostic  aids.  Weissmuller,  Barton,  and  Rogers  (1974a) 
cover  source  programs  for  the  UNIVAC  1108  version  of  CODAP.  Included  are  34  FORTRAN 
main  programs  and  78  FORTRAN  and  20  Assembly  Language  subroutines.  Weissmuller, 
Barton,  and  Rogers  (1974b)  discuss  programmer  notes  on  100  library  subroutines  used 
by  the  UNIVAC  1108  version  of  CODAP.  Arranged  in  alphabetical  order  by  subroutine 
name,  each  note  includes:  (a)  comments  on  peculiar  requirements  for  conversion  and 

use  of  subroutines;  (b)  summaries  of  subroutine  functions  and  a list  of  entry 
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points;  and  (c)  for  each  entry  point,  a calling  sequence  with  descriptions  of 
input  and  output  arguments. 

Typical  uses  of  the  CODAP  system  for  course  developers  are  given  in  the  next  two 
reports.  First,  the  job  inventory  method  of  job  analysis  for  officer  positions  was 
studied  by  Mayo,  Nance,  and  Shigekawa  (1974).  The  authors  concluded  that  the  in- 
ventory method  was  operationally  adequate  but,  due  to  the  officers'  broader, 
concept-oriented  responsibilities,  more  specific  approaches  to  task  construction  are 
required  than  are  usually  needed  for  airmen  positions.  To  resolve  the  issues  of 
task  specificity  and  breadth  of  coverage  for  officers'  fields,  "heavy  front-end 
work"  (i.e.,  research  requisite  information)  is  recommended. 

Second,  ski  11 -level  career  ladders  in  Accounting  and  Disbursement  were  studied 
by  Stacey  and  Hazel  (1974).  More  than  1200  Accounting  and  Finance  Supervisors 
examined  254  journeyman-level  tasks  and  rated  the  importance  of  each  task  relative 
to  satisfactory  supervisory  performance.  Job  analyses  of  career  work  experiences 
indicated:  (a)  many  supervisors  had  limited  career  experience  in  the  desirable 
journeyman- level  tasks;  (b)  many  supervisors  had  spent  most  of  their  career  time  in 
very  few  of  eight  accounting  and  finance  work  areas;  and  (c)  lack  of  supervisory 
work  experience  was  related  to  on-the-job  performance.  The  authors  concluded  that 
the  present  accounting  and  finance  work  management  unit  was  too  broad  for  required 
work  activities,  and  recommended  the  separation  of  the  Accounting  (671x1)  and  Dis- 
bursement (671x3)  career  ladder  through  the  seven-skill  level. 

In  a recent  paper,  Christal  and  Weissmuller  (1975)  describe  five  new  programs 
added  to  the  UNIVAC  1108  version  of  CODAP  which  are  designed  to  analyze  and  manipu- 
late information  describing  work  tasks  rather  than  jobs  or  persons.  One  use  of  the 
new  program  is  to  rank-order  training  priorities  for  curriculum  designers.  Although 
the  technical  discussion  is  beyond  the  intent  of  this  bibliography,  the  authors  re- 
port that  control  card  specifications  are  now  available  which  can  be  reprogrammed 
for  execution  on  IBM  360-370  series  computers. 
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DATA  SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key  Number: 
Authoring  aid 


Computer- based 
Computer-based  job  analysis 
Extensive  field-tests 
Military  (Air  Force) 

1.1  Analyze  Job 
type:  Tool 
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REFERENCE  NUMBER:  1.1.2 

ANALYZE  JOB 

CITATION: 

Foley,  J.  P.,  Jr.  ' Task  analyses  for  job  performance  aids  and  related  training.  In 
T.  C.  Rowan  (Ed.),  Proceedings  of  the  Invitational  Conference  On  Improved 
Information  Aids  for  Technicians.  Sumner,  Maryland:  Logistics  Management 

Institute,  May  1975. 

ABSTRACT: 

A major  problem  in  educating  instructional  personnel  in  the  use  of  task  analysis 
has  been  a lack  of  agreement  as  to  its  purpose  and  method  of  execution.  Emphasizing 
maintenance,  Foley  argues  that  perhaps  the  more  important  purpose  of  task  analysis  is 
to  determine  the  content  of  training,  and  offers  a standardized  execution  methodology. 

The  first  step  in  task  analysis  is,  of  course,  identification  of  tasks  actually 
done  on  the  job.  If  job  requirements  are  being  done  only  partially,  then  what  peo- 
ple should  do  should  also  be  identified.  Work  sheets  for  a Task  Identification 
MatVix  (TIM)  are  given  which  list  each  system  hardware  component  and  its  parts  on  a 
numbered  line.  Column  headings  on  the  TIM  consist  of  14  maintenance  actions  such 
as  adjust,  align,  calibrate,  checkout,  handle,  clean,  lubricate,  and  inspect.  Then, 
for  each  cell  in  the  matrix,  a code  is  entered  which  indicates  the  level  of  mainte- 
nance required.  After  every  task  has  been  identified  and  scoped,  a Task  Inventory 
is  produced  which  is  simply  a listing  and  numbering  of  each  task  identified  in  the 
matrix.  The  computer  storage  and  handling  of  task  information  would  permit  auto- 
matic generation  of  task  inventories.  (See  references  for  the  CODAF  System,  1.1.1) 

The  second  step  in  task  analysis  is  functional  analysis  of  the  identified  tasks. 
Under  the  umbrella  of  a Task  Description  Index  and  Management  Matrix  (TDIMM),  the 
functional  analysis  information  is  formatted  on  three  worksheets.  The  preliminary 
Information  Worksheet  lists  the  hardware  item  with  which  the  task  is  associated  as 
well  as  required  supplies,  forms,  replacement  parts  and  technical  orders,  and  neces- 
sary notes,  cautions,  and  warnings.  The  Task  Description  and  Information  Index 
merges  this  information  with  summary  Information  about  the  task  such  as  criticality, 
frequency  of  occurrence,  number  of  steps,  and  task  type.  The  Detailed  Step  Descrip- 
tion Worksheet  provides  for  a detailed,  step-by-step  analysis  of  the  task.  It  in- 
cludes maintenance  support  information  and  special  instructions.  When  a task 
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involves  use  of  test  equipment  and  special  tools,  a Test  Equipment  and  Tool  Use 
Form  is  completed.  Graphics  of  each  form  are  given  in  the  text. 

Using  these  documents  does  not  ensure  that  effective  directions  are  finally 
written  for  the  maintenance  technician.  A third  step  is  required  which  involves 
"a  software  middle  man"  who  is  able  to  add  the  proper  behavioral  cues  to  the  techni 
cal  directions  and  analysis.  For  example,  in  one  application  where  exploded  views 
of  an  aircraft  did  not  supply  the  required  pictorial  cues  for  disassembly,  Foley 
says  that  it  was  necessary  to  add  the  phrase  "when  lens  is  closed  it  looks  shiny." 

A variant  of  conventional  task  analysis  is  special  analysis  for  troubleshooting 
tasks.  Special  guidance  must  be  given  both  to  task  analyzers  and  technicians. 

Foley  recommends  the  use  of  Maintenance  Dependency  Charts,  Logic  Trees,  and  Fully 
Proceduralized  Job  Performance  Aids,  details  of  which  are  beyond  the  scope  of  his 
report. 

Although  the  author  used  maintenance  examples  to  illustrate  the  utility  of 
standardized  worksheets,  it  should  not  be  difficult  to  adapt  the  forms  for  use  in 
other  task  analysis  situations. 

DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  Number: 

Authoring  aid  type: 

Number  of  pages: 

Number  of  references: 


Print 

Manual  task  analysis 
None 

Military  (Anny) 

1.1  Analyze  Job 
Tool 
21 
31 
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REFERENCE  NUMBER:  1.1.3 

ANALYZE  JOB 

CITATION: 

Task  Inventory  Exchange.  Brochure. 

Columbus: 

The  Ohio  State  University,  Center 

for  Vocational  Education,  1975. 


ABSTRACT 

Task  analysis  is  a time-  and  money-consuming  activity  which  generates  task 
inventories  for  specific  jobs  in  organizations  or  occupational  areas  with  a limited 
intent  of  use  on  the  part  of  their  developers.  However,  if  their  existence  and 
character  are  known  these  same  job  inventories  may  have  much  wider  application.  To 
bridge  the  gap  between  those  who  need  but  are  unaware  of  existing  task  inventories 
and  those  who  have  previously  developed  them,  the  Task  Inventory  Exchange  (TIE)  was 
created.  The  TIE  has  four  functions:  (a)  identify  existing  occupational /task  in- 
ventories; (b)  collect,  catalog,  and  store  existing  task  inventories;  (c)  publish 
directories  of  available  task  inventories;  and  (d)  supply  microfiche  or  hard  copies 
of  available  task  inventories. 

The  TIE  obtains  task  inventories  through  distribution  of  a brochure  and  place- 
ment of  advertisements  in  eight  professional  journals.  Contributions  of  task  in- 
ventories and  related  literature  for  the  TIE  should  be  made  in  two  copies.  It  also 
actively  conducts  comprehensive  literature  searches  in  the  data  bases  of  Educational 
Resources  Information  Centers  (ERIC),  National  Technical  Information  Service  (NTIS), 
Abstracts  of  Instructional/Research  Materials  in  Vocational  and  Technical  Education 
(AIM/ARM),  and  University  Microfilms.  Results  of  these  activities  are  collected  and 
stored  on  magnetic  tape  and  in  hard-copy  volumes. 

The  TIE  periodically  publishes  directories  of  task  inventories.  Volume  1, 
Directory  of  Task  Inventories,  was  published  in  1974  and  contained  citations  for 
77  publications  listing  task  inventories  for  280  job  titles  or  occupational  areas. 
Volume  2,  same  title,  was  published  in  1975  and  contained  citations  for  177  publica- 
tions listing  inventories  for  379  occupations  or  jobs.  Volume  3 is  currently  in 
press.  The  cost  for  Volumes  1 and  2 is  $4  and  $7,  respectively. 

The  TIE  supplies  partial  or  whole  task  inventories  on  a cost-recovery  basis. 

The  first  five  microfiche  of  any  one  document  are  65  cents  and  each  additional 
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fiche  is  15  cents.  Paper  copies  of  task  inventories  are  five  cents  a page. 


Further  information  concerning  the  TIE  may  be  obtained  from: 

Paul  E.  Schroeder,  Project  Director 

Task  Inventory  Exchange 

Center  for  Vocational  Education 

The  Ohio  State  University 

1960  Kenny  Road 

Columbus,  Ohio  43210 

(614)  486-3655. 


DATA  SUMMARY: 
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Microfiche,  Print 

Job  Analysis 

None 

Civilian 

1.1  Analyze  Job 

Tool 

Approximately  20 
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REKERENCE  NUMBER:  1.2.1 

SELECT  TASKS/FUNCTIONS 

CITATION: 

Cline,  J.  A.  A multivariate  statistical  model  to  predetermine  preferable  aircraft 
assignments:  A feasibility  study  (Doctoral  Dissertation,  Arizona  State  Univer- 
sity, 1973).  University  Microfilms  No.  73-20,  494. 

ABSTRACT: 

If  job  assignments  can  be  made  before  students  begin  or  complete  training,  it 
is  possible  to  identify  appropriate  training  tasks  or  functions  that  are  needed  for 
job  assignment  and  tailor  individual  learning  programs  for  individual  students. 

Cline  developed  a model  for  selecting  pilots  for  end  aircraft  assignment  (advanced 
pilot  training)  early  in  Undergraduate  Pilot  Training.  His  technique  may  be  ex- 
tended to  other  fields. 

Peer-ranking  data  were  obtained  from  an  earlier  study  and  used  as  the  primary 
operational  measure  of  a pilot's  success  flying  an  operational  aircraft.  Secondary 
measures  included  advanced  pilot  training  grades,  and  on-the-job  Officer  Effective- 
ness Ratings  by  supervisors.  Two  techniques  used  in  the  study  are  of  interest  to 
course  developers:  (a)  conversion  of  peer-rankings  into  ordinal  data j and  (b)  selec- 
tion of  subgroups  for  classes  of  training. 

One  disadvantage  of  peer  ranking  is  that  while  it  is  possible  to  determine  who 
is  first,  second,  or  third,  it  is  not  possible  to  measure  the  amount  of  difference 
between  the  first  and  the  second,  and  between  the  second  and  the  third.  Therefore, 
Cline  converts  the  peer  rankings  into  ordinal  data  by  using  a technique  similar  to 
Thorndike's  T-Score  procedure.  The  ordinal  data  become  the  criteria  for  the  further 
selection  of  subgroups  for  classes  of  training.  The  validity  of  the  criteria  are, 
Cline  admits,  subjective  and  "cannot  be  statistically  measured",  but  their  relia- 
bility was  measured  by  the  coefficient  of  concordance  (W),  which  is  a measure  of 
the  commonality  of  judgements.  In  this  particular  study,  it  indicated  the  degree 
to  which  pilots  kept  the  same  relative  standing  in  the  group  when  peers  ranked  all 
members  of  the  group. 

Selection  of  subgroups  for  classes  of  training  was  based  upon  selected  variables 
such  as  military  grade  and  flying  hours  in  jet-powered  aircraft,  and  upon  a 
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statistical  technique  called  Automatic  Interaction  Detector  (AID).  Developed  and 
refined  by  Air  Force  personnel,  the  AID  technique  essentially  divides  the  total 
sample  Into  mutually  exclusive  subgroups  of  Increased  homogeneity  with  respect  to 
the  dependent  variable.  For  example,  using  AID,  Cline  found  that  the  most  Important 
variable  for  explaining  the  variance  in  pilot  Armed  Forces  Qualifying  Test  (AFQT) 
was  education.  The  first  split  separated  college  graduates  from  high  and  non-high 
school  graduates  and  accounted  for  68  percent  of  the  variance.  The  second  split 
separated  high  and  non-high  school  graduates  and  accounted  for  an  additional  nine 
percent  of  the  variance. 

Finally,  the  author  combined  regression  analysis  and  discriminant  analysis  to 
form  a classification  model  which  establishes  group  membership  and  Indicates  the 
probable  degree  of  Individual  success  In  such  a group.  Decision  functions  match 
Individuals  to  groups.  When  training  requirements  for  groups  such  as  "fighter 
pilot"  or  "bomber  pilot"  or  "navigator"  are  identified  (which  Cline  does  not  do), 
then  tailored  learning  programs  for  individuals  within  those  groups  can  be 
developed. 

DATA  SUMMARY: 


Media: 

Computer-based 

Application: 

Individualized  Instruction 

Evaluation: 

None 

Environment: 

Military  (Air  Force) 

Key  Number: 

1.2  Select  Tasks/Functions 

Authoring  Aid  Type: 

Procedure 

Number  of  Pages: 
Number  of  References: 

211 
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REFERENCE  NUMBER:  1.2.2 

SELECT  TASKS/FUNCTIONS 

CITATION: 

McKnight,  A.  J.,  & Adams,  B.  B.  Driver  education  task  analysis.  Volume  II:  Task 
analysis  methods.  Final  report  (HumRRO  Tech.  Rep.  72-13).  Alexandria,  Va.: 
Human  Resources  Research  Organization,  November  1970.  (ERIC  Document  Reproduc- 
tion Service  No.  ED  075  624) 

ABSTRACT: 

During  the  development  of  performance-oriented  driver  education  objectives  and 
a set  of  measuring  devices  for  evaluating  attainment  of  the  objectives,  over  one 
thousand  behavioral ly-relevant  system  characteristics  were  identified.  The  behav- 
iors identified  during  the  analysis  of  driving  tasks  varied  considerably  in  their 
criticality  to  the  safety  and  efficiency  with  which  the  highway  traffic  system 
operates.  Since  an  efficient  program  of  driver  education  must  account  for  this 
variation,  an  evaluation  of  behavior  criticality  became  a part  of  the  analysis. 

The  approach  to  criticality  evaluation  was,  of  necessity,  a judgemental  one;  there 
were  insufficient  data  to  support  any  empirical  determination  of  criticality.  A 
group  of  authorities  in  the  area  of  traffic  safety  — driver  educators,  enforcement 
officers,  license  officials,  and  fleet  safety  personnel  — performed  the  evaluation. 
The  plan  called  for  each  evaluator  to  rank,  in  terms  of  criticality,  three  groups 
of  25  behaviors  from  a total  list  of  1500  behaviors.  This  allowed  each  behavior  to 
be  ranked  five  times.  However,  since  seven  of  the  evaluators  did  not  return  com- 
pleted evaluations,  the  number  of  rankings  in  some  cases  was  four  instead  of  five. 
The  rankings  were  then  transformed  to  standard  scores  with  a mean  of  zero  and  a 
standard  deviation  of  10.  The  scores  for  each  behavior  were  averaged  to  obtain  a 
criticality  index  for  each  behavior.  An  analysis  of  variance  showed  a high  degree 
of  agreement  among  the  evaluators. 

There  were  four  steps  in  the  criticality  evaluation:  (a)  obtaining  criticality 
data;  (b)  selecting  an  evaluation  method;  (c)  administering  the  evaluation;  and 
(d)  generating  criticality  indices. 

Data  were  obtained  from  a literature  review  and  from  analysis  of  1100  actual 
automobile  accidents.  An  evaluation  method  was  chosen  after  a pilot  study  of  three 
approaches  --  rating,  task  group  ranking,  and  random  group  ranking  — found  that 
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random  group  ranking  provided  the  only  straightforward  means  of  comparing  behaviors 
across  tasks.  Once  the  data  had  been  obtained  and  the  evaluation  method  had  been 
chosen,  the  actual  administration  of  the  evaluation  materials  began.  One  hundred 
evaluators  chosen  with  the  aid  of  the  National  Highway  Safety  Bureau  were  mailed  a 
sample  of  25  behaviors,  the  task  descriptions,  and  a set  of  directions.  Upon  com- 
pleting a ranking,  the  evaluator  was  asked  to  place  a colored  card  in  the  stack  of 
ranked  behaviors  beneath  the  last  behavior  he  felt  was  critical  enough  to  be  re- 
quired of  qualified  drivers.  The  rankings  were  then  returned  to  the  authors. 

The  data  from  94  evaluators  were  used  to  generate  criticality  indices.  In 
analyzing  the  results  of  the  rankings,  individual  ranks  were  converted  to  their 
normalized  equivalents.  The  normalized  scores  ranged  from  +2.0  for  a rank  of  "1" 
to  -2.0  for  a rank  of  "25,"  and  the  mean  of  zero  corresponded  to  a rank  of  13.  The 
means  of  these  normalized  values  and  their  associated  standard  deviation  were  then 
calculated  for  each  of  the  1500  behaviors. 

Since  the  behaviors  were  not  a representative  sample  of  the  population,  their 
relative  standing  to  one  another  would  not  approximate  their  standing  within  the 
population,  and  so  simply  ranking  the  behaviors  against  one  another  was  not  appro- 
priate. Thus,  an  additional  procedure,  described  below,  was  needed  to  create  valid 
criticality  indices. 

First,  a group  of  the  previously  ranked  behaviors  was  selected  to  provide  a 
behavior  whose  mean  rank  corresponded  to  each  of  the  25  ranks.  Next,  an  additional 
group  of  five  evaluators  took  each  of  the  unranked  behaviors  and  assigned  it  the 
rank  of  the  behavior  in  the  list  of  the  ranked  25  behaviors  to  which  it  most  closely 
approximated  <n  criticality.  The  behavior  was  then  given  the  same  rank  value  as  the 
corresponding  ranked  behavior.  The  evaluator  was  also  asked  to  decide  whether  the 
behavior  was  sufficiently  critical  to  be  considered  a minimum  requirement  for  a 
qualified  driver.  The  results  were  then  processed  in  the  same  manner  as  described 
for  the  previously  ranked  behaviors. 

Although  the  criticality  indices  remain  ultimately  subjective  and  one's  confi- 
dence in  their  validity  rests  with  his  confidence  in  the  evaluators,  a high  level 
of  agreement  supports  confidence  while  non-agreement  destroys  it.  An  analysis  of 
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variance  indicated  a high  level  of  agreement  among  evaluators  concerning  the  criti- 
cal behaviors.  Further,  an  analysis  of  variance  of  inter-task  differences  indicated 
much  similarity  in  the  criticality  of  behaviors  within  a particular  task.  The  re- 
sults of  each  analysis  of  variance  are  given  in  the  text  of  the  report. 


Although  the  procedure  suggested  by  McKnight  and  Adams  is  rather  involved, 
especially  where  large  numbers  of  behaviors  are  considered,  it  provides  a solution 
to  the  course  developer's  question  of,  "What  tasks  are  sufficiently  critical  that 
they  must  be  taught?",  and  thereby  aids  the  selection  of  appropriate  tasks  and 
functions. 


Four  appendices  are  given:  A.  Behaviorally  Relevant  System  Characteristics; 
B,  Directions  to  Evaluators  of  Task  Criticality;  C.  Task  Criticality  Means  and 
Standard  Deviations;  and,  D.  List  of  Evaluators. 


DATA  SUMMARY: 
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Number  of  References: 


None 

Individualized  Instruction 

None 

Civilian 

1.2  Select  Tasks/Functions 

Procedure 

45 

None 


B-19 


AUTHORING  AID 

MDC  El 570 

BIBLIOGRAPHY 

VOLUME  1 

1 1 FEBRUARY  1977 

REFERENCE  NUMBER:  1.2.3 

SELECT  TASKS/FUNCTIONS 

CITATION: 

Powers,  L.  The  systems  approach  to  functional  job  analysis:  Task  analysis  of  the 

physician's  assistant.  Volume  1:  Task  analysis  methodology  and  techniques. 

Winston-Salem,  North  Carolina:  Wake  Forest  University,  Bowman  Gray  School  of 

Medicine,  1971.  (ERIC  Document  Reproduction  Service  No.  ED  D59  378) 

ABSTRACT: 

The  basic  structure  of  a task  statement  has  two  parts:  the  end  result;  and  the 

worker  activity  required  to  bring  about  the  end  result.  The  end  results  are  those 
steps  which  must  be  carried  out  to  accomplish  the  organization's  objectives.  For 
physician's  assistant,  typical  end  results  might  be  "progress  notes  on  hospital, 
home,  and  office  patients"  and  "hospital  discharge  summaries  for  physician's  review 
and  signature."  Typical  work  activities  for  the  same  profession  include  "perform 
and/or  schedule"  and  "assist  the  physician  with...."  These  are  basic  structures 
which  every  competent  task  statement  should  contain. 

Powers,  however,  has  added  a phrase  to  the  task  statements  collected  in  this 
volume  by  which  task  statements  may  aid  in  selection  of  tasks  and  functions  for 
training.  The  phrase  is:  in  order  to.  The  phrase  may  be  used  to  indicate  an  in- 
tended outcome  of  a task,  and  thereby  group  students  according  to  r<equired  levels 
of  competence.  Those  students  with  nearly  adequate  competence  to  do  the  intended 
task  should  receive  less  training  than  those  who  lack  a basic  competence.  Consider, 
for  example,  two  tasks  in  electronics  training  which  have  different  outcomes  but  a 
same  worker  activity.  The  tasks  are  "troubleshooting  an  oscilloscope"  and  "using 
an  oscilloscope,’  and  the  worker  activity  is  "turn  on  an  oscilloscope."  If  a stu- 
dent turns  on  an  oscilloscne  in  order  to  troubleshoot  it  rather  than  in  order  to 
operate  it,  a very  different  level  of  training  — both  in  content  and  extent  --  is 
required  for  the  end  result.  Also,  students  may  be  expected  to  bring  different 
prior  abilities  to  each  task. 

The  collection  of  task  statements  for  physician's  assistant  was  developed  dur- 
ing a systematic  sampling  of  the  professional  activities  of  pediatricians,  family 
practitioners,  surgeons,  obstetricians,  and  internists.  Course  outlines,  training 
objectives,  and  goals  for  training  physician's  assistants  were  also  developed. 
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REFERENCE  NUMBER:  1.3.1  CONSTRUCT  JOB  PERFORMANCE  MEASURES 

CITATION: 

Gael,  S.  Employment  test  validation  studies.  JSAS  Catalog  of  Selected  Documents 

in  Psychology.  1974,  4,  95  (Ms.  No.  7J1) 

ABSTRACT: 

Gael  reports  a study  of  five  Bell  System  jobs  in  order  to  identify  job  activi- 
ties and  dimensions,  job  products  and  factors  for  product  quality,  relative  impor- 
tance of  various  products  and  job  activities,  and  the  skills  required  to  perform 
the  work.  The  activity  analysis  results  — "activity  analysis"  being  a Bell  System 
word  substituted  for  "task  analysis"  — were  translated  into  employment  tests  and  job 
performance  measures.  Gael's  discussion  is  confined  to  the  selection  and  design  of 
the  instruments  used  to  gauge  aptitudes  and  proficiencies.  It  does  not  deal  with 
results. 

Five  Bell  System  jobs  were  studied:  (1)  Service  Representative;  (2)  Toll  Opera- 
tor; (3)  Directory  Assistance  Operator;  (4)  Traffic  Service  Position;  and  (5) 
Clerical.  As  an  example  of  the  thoroughness  and  adaptability  of  the  Bell  System 
studies,  the  Service  Representative  (SR)  and  the  Clerical  Tests  will  be  discussed. 

SR  work  positions  consist  of  a special  desk  in  which  2500  to  4000  customer  records 
are  filed;  a position  identification  board  that  identifies,  by  telephone  number, 
all  the  records  filed  in  the  desk;  a telephone;  several  reference  books;  an  elabor- 
ate in-out  basket  for  work  status  documentation;  and  other  incoming  and  outgoing 
paper  work.  SR  activities  consist  of  customer  contact,  clerical,  computational, 
and  filing  efforts  which  are  performed  under  a steady  influx  of  customer  and  intra- 
company calls.  A call  frequency  analysis,  however,  shows  that  the  four  most  fre- 
quent contacts  require:  (a)  taking  orders  for  new  telephone  service  or  changes  to 
existing  service;  (b)  explaining  and  adjusting  telephone  bills;  (c)  recording  equip- 
ment malfunctions  or  service  problems  and  reporting  them  to  maintenance;  and  (d) 
notifying  customers  of  overdue  accounts  and  arranging  payment  schedules. 

Two  SR  job  performance  measures  were  developed  to  measure  acquisition  and 
application  of  job  knowledge  and  skills,  and  to  measure  job  proficiency  in  a 
standardized  situation.  The  primary  purpose  of  the  written  Job  Knowledge  Review 
(JKR)  was  to  determine  the  comprehension  level  of  company  policies,  and  job 
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procedures  and  practices.  Using  multiple-choice  and  constructed-response  questions, 
the  JKR  covered  topics  such  as  SR  reference  materials,  types  of  equipment  available 
to  customers,  charges  for  individual  pieces  or  combinations  of  equipment  and  ser- 
vices, customer  complaints,  and  intercompany  and  intracompany  communications.  The 
JKR  was  administered  to  a class  of  SRs  at  the  end  of  formal  training  and  required 
about  one  hour  to  complete.  On  the  other  hand,  the  Job  Performance  Review  (JPR) 
was  a job  proficiency  measure  involving  simulation.  One  administrator  initiated 
calls  to  the  SR  in  a time-  and-setting-controlled  situation  while  another  specially- 
trained  administrator  listened  to  each  "customer"  contact  and  evaluated  the  SR's  oral 
performance.  The  JPR  consisted  of  contact-generated  and  other  clerical  work  and  re-' 
quired  75  to  90  minutes  to  complete.  Three  JPR  scores  were  given  for  each  simula- 
tion: (a)  Record  Preparation  (RP),  the  sum  of  the  points  accorded  each  part  of  a 
record  prepared  or  completed  as  determined  by  comparison  with  a model  set  of  re- 
cords with  points  already  assigned;  (b)  Verbal  Contact  (VC),  the  sum  of  points 
assigned  each  part  of  the  verbal  interaction  with  the  "customer;"  and  (c)  Filing  (F), 
the  number  of  records  not  in  the  designated  location  when  the  JPR  was  terminated. 

The  JKR  and  the  three  JPR  scores  were  combined  in  a composite  score  that  repre- 
sented the  acquisition  of  knowledge  and  the  ability  to  apply  that  knowledge  on  the 
job.  In  response  to  management  emphasis  on  customer  satisfaction,  the  Verbal  Con- 
tact (VC)  score  was  weighted  twice  that  of  the  other  components. 

The  paradigm  described  for  the  SR  test  was  followed  for  the  three  Operator  test 
studies  with  only  minor  modifications. 

Nearly  120,000  telephone  company  employees  work  at  400  jobs  classified  as 
clerical.  The  main  problem  in  developing  a job  performance  measure  for  clerical 
workers,  then,  was  to  reduce  systematically  clerical  jobs  to  activities  common  to 
most  clerical  jobs.  Descriptions  of  400  jobs  were  reduced  to  eight  dimensions  and 
work  samples  in  each  dimension  were  obtained  which,  after  refinement  and  printing, 
resulted  in  a Clerical  Performance  Measurement  (CPM)  package  and  an  administrative 
manual  used  to  train  CPM  administrators  in  giving  and  scoring  the  CPM. 

The  administration,  media,  and  format  of  the  CPM  could  perhaps  be  readily 
adapted  to  a clerically-oriented  Defense  Occupational  Specialty  (DOS)  such  as  the 
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Air  Force's  Inventory  Management  career  field,  below  the  seven-skil li level . 
Administration  of  the  CPM  requires  about  two  days  and  can  be  given  to  groups  of 
10  people  by  one  administrator.  In  addition  to  the  CPM  materials  and  administra- 
tive manuals,  other  requirements  include  tape  cassettes,  playback  units,  office 
supplies,  a timer,  and  an  enclosed,  quiet  room  that  can  hold  up  to  10  study  partici- 
pants at  separate  tables.  Completed  CPM  packages  are  scored  by  the  administrator 
and  independently  by  an  assistant. 

The  eight  CPM  packages  consist  of  five  timed  clerical  activities,  three  self- 
paced  programmed  texts,  and  an  audio  tape.  Each  unit  is  contained  in  a separate 
folder  placed  inside  an  expansion  folder.  The  timed  clerical  activities,  completed 
in  accordance  with  specific  instructions,  are  Filing,  Classifying,  Posting,  Checking 
and  Coding.  The  self-paced  units  contain  programmed  texts  and  examinations  that  are 
parts  of  the  telephone  company  training  course  for  clerks.  The  examinations  are  in- 
tended to  measure  the  ability  to  learn  the  specific  content  and  detail  of  clerical 
jobs.  They  are  entitled  Toll  Fundamentals,  Punched  Card  Fundamentals,  and  Plant 
Repair  Service.  Job  proficiency  measures  are  obtained  by  combining  the  eight  CPM 
scores  for  each  study  participant. 


The  five  employment  test  validation  studies  discussed  above  covered  jobs  in 
which  about  300,000  Bell  System  employees  work,  and,  according  to  Gael,  many  more 
than  that  are  screened  yearly  for  those  jobs. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  Number: 

Authoring  Aid  Type: 
Number  of  Pages 
Number  of  References: 


Simulation,  programmed  text,  and  audio  tape 
Self-paced  job  performance  measurement 
None 

Civilian 

1.3  Construct  Job  Performance  Measures 
Procedure 
8 
5 
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REFERENCE  NUMBER:  1.4.1 

ANALYZE  EXISTING  COURSE 

CITATIONS: 

Inventory  management  specialist  and  inventory  management  supervisor.  Specialty 
training  standard  645X0.  Washington,  D.C.:  Department  of  the  Air  Force, 

Headquarters,  11  August  1975. 

Materiel  facilities  specialist.  Plan  of  instruction  3ABR64730-1.  Lowry  Air  Force 
Base,  Colorado:  Lowry  Technical  Training  Center,  School  of  Applied  Aerospace 

Sciences,  Department  of  Logistics  Training,  2 January  1974. 

Materiel  facilities  specialist.  Course  chart  3ABR64730-1.  Lowry  Air  Force  Base, 
Colorado:  Lowry  Technical  Training  Center,  School  of  Applied  Aerospace 

Sciences,  Department  of  Logistics  Training,  12  September  1973. 

ABSTRACT: 

Analysis  of  an  existing  Air  Force  course  appears  in  the  Specialty  Training 
Standard  (STS),  Plan  of  Instruction  (POI),  and  the  Course  Chart.  These  are 
official.  Air  Force-prepared  documents  intended  to  provide  guidance  to  course 
administrators,  instructors,  and  developers. 

The  STS  describes  in  terms  of  a number-letter  code  the  knowledges  and  skills  an 
airman  in  a particular  career  field  (such  as  Inventory  Management)  must  have  by  end 
of  training.  The  number  codes  indicate  required  task  performance  levels,  and  range 
from  1 (least)  to  4 (complete  task  proficiency).  The  letter  codes  are  both  lower 
and  upper  case,  and  indicate  task  or  subject  knowledge  levels,  respectively.  Lower 
case  letter  codes  range  from  "a"  (knowledge  of  nomenclature)  to  "d"  (knowledge  of 
theory).  Upper-case  letter  codes  range  from  "A"  (knowledge  of  facts)  to  "D"  (knowl 
edge  of  evaluation).  The  requirements  for  task  knowledges  and  skills  are  given  by 
skill  level.  For  example,  one  task  in  the  Inventory  Management  career  field  is 
"conducting  inventories",  which  three-level  airmen  are  required  to  know  to  the  "A" 
STS  level  and  five-level  airmen  are  required  to  know  to  the  "B"  STS  level.  That 
means  a three-level  airmen  "can  identify  basic  facts  and  terms  about  the  subject" 
while  five-level  airmen  "can  explain  relationship  of  basic  facts  and  state  general 
principles  about  the  subject."  The  STS  is  useful  in  determining  a number  of  task- 
related  Issues  such  as  selection  of  training  tasks  or  writing  behavioral  objectives 
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because  it  is  an  officially  accurate  analysis  of  required  knowledges  and  skills  in 
a particular  field. 

The  Plan  of  Instruction  (POI)  is  a publication  that  identifies  units  of  instruc- 
tion for  a particular  course  and  indicates  the  following  for  each  unit  of  instruc- 
tion: (a)  duration  of  instruction  and  the  day  on  which  it  is  presented;  (b)  the 

criterion  levels  to  be  reached;  (c)  support  materials;  and,  (d)  the  STS  reference 
for  the  course  content  being  presented. 


The  Course  Chart  divides  the  course  into  blocks  of  instruction  and  specifies 
the  sequence  in  which  the  lessons  are  presented.  Major  equipment  items  used  in  a 
course  are  identified  and  the  number  of  hours  of  technical  and  related  training  is 
specified.  Related  training  includes  commander's  calls  and  briefings,  traffic 
safety,  and  human  relations. 


DATA  SUMMARY: 


Media: 

Print 

Application; 

Conventional  and  self-paced  instruction 

Evaluation: 

Quality  control  by  experienced  personnel  in  the  career 
field 

Environment: 

Military  (Air  Force) 

Key  Number: 

1.4  Analyze  Existing  Course 

Authoring  Aid  Type: 

Tool 

Number  of  Pages: 

Variable  by  career  field 

Number  of  References: 

Variable  by  career  field 
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REFERENCE  NUMBER:  1.4.2 

CITATION: 

Lewis,  W.  E.,  Willow,  J.  D.,  Brock,  G.  R.,  Lonigro,  J.  K. , Jr.,  Eschenbrenner,  A.  J., 
& Hanson,  A.  L.  Precision  measuring  equipment  (PME)  individualized  instruction 
(Report  No.  AFHRL-TR-74-46).  Lowry  Air  Force  Base,  Colorado;  Air  Force  Human 
Resources  Laboratory,  Technical  Training  Division,  August  1974. 

ABSTRACT: 

A portion  of  an  Air  Force  course  in  Precision  Measuring  Equipment  (PME) 
(3ABR32430-2)  was  converted  from  conventional  instruction  to  self-paced  individ- 
ualized instruction  in  order  to  determine  the  feasibility  of  individualizing  the 
entire  course  as  part  of  the  Advanced  Instructional  System  (AIS).  Self-paced  pro- 
grammed texts  and  audio-visual  materials  were  developed,  administered,  and  evaluated 
for  a 90-hour  block  of  instruction.  Training  time  was  reduced  by  31%  with  all 
trainees  achieving  criterion  objectives. 

One  phase  of  the  study  concerned  analysis  of  the  existing  course,  which  was 
done  in  five  tasks.  The  first  task  was  to  select  course  segments  for  individuali- 
zation according  to  five  selection  criteria.  Chosen  course  segments  should: 

(a)  sample  the  full  range  of  skills  and  knowledges  required  in  the  PME  course; 

(b)  sample  the  full  range  of  media  requirements  in  the  PME  course;  (c)  contain 
skills  and  knowledges  which  are  prerequisites  for  accomplishing  frequently  occur- 
ring job  tasks;  (d)  contain  skills  and  knowledges  essential  to  both  current  and 
projected  frequently  occurring  job  tasks;  and,  (e)  be  able  to  be  implemented  with- 
out disrupting  normal  trainee  flow  through  non-selected  segments. 

The  second  task  was  to  analyze  course  and  job  documentation.  The  following 
doucments  were  analyzed:  (2)  Course  Chart;  (b)  Specialty  Training  Standard  (STS); 

(c)  Plan  of  Instruction  (POI);  (d)  student  study  guides,  workbooks,  programmed 
texts,  and  handouts;  and,  (e)  the  Occupational  Survey  Report  (OSR).  The  Course 
Chart,  POI,  and  STS  proved  useful  in  providing  an  overview  of  the  course  (see 
annotations  for  these  documents  in  this  section,  1.4.1)  while  the  OSR  was  used  to 
obtain  data  on  task,  frequency  of  performance,  and  total  time  spent  doing  the  task 
on  the  job. 
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The  third  task  was  to  choose  the  target  population.  Several  factors  were  con- 
sidered: (a)  size  and  rate  of  increase  or  decrease  of  target  population;  (b)  loca- 
tion and  size  of  instructional  "units";  (c)  Airman  Qualification  Examination  (AQE) 
scores  relevant  to  the  general  and  specific  knowledges  and  skills  required  by  the 
PME  course;  (d)  scores  on  standardized  national  achievement  tests  and/or  on  vali- 
dated local  achievement  tests;  (e)  entry  skills  and  learning  rates;  and,  (f)  appro- 
priate instructional  strategies. 

The  fourth  task  was  to  examine  the  training  environment.  The  PME  laboratory 
contained  12  classrooms.  Those  classrooms  devoted  to  academic  topics  (such  as 
basic  electronic  theory)  contained  tables  and  chairs  while  classrooms  given  to 
"hands-on"  practical  performance  contained  not  only  tables  and  chairs  but  also 
workbenches  and  electronic  equipment.  The  primary  pieces  of  equipment  in  the  PME 
course  were  calibration  standards  and  applied  measurement  trainers.  Since  the 
standards  and  trainers  were  available  in  limited  quantities,  trainees  in  some 
segments  of  instructions  worked  in  pairs. 

The  fifth  task  was  to  select  representative  segments  of  the  PME  course  for 
individualization.  Air  Training  Command  specified  that  Block  X,  Waveform  Analysis, 
be  individualized,  which  satisfied  selection  criteria  two  through  five.  An  addi- 
tional 24  hours  of  instruction  were  selected  from  other  blocks  in  order  to  satisfy 
selection  criterion  1. 


Eight  appendices  are  included  in  the  report:  A.  Sample  Task  Analysis;  B.  Sam- 
ple Learning  Objective;  C.  Lesson  Flowchart;  D.  Trainee  Progress  Flowchart; 

E.  Instructional  Method  Questionnaire;  F.  Block  X,  Small-Group  Tryout  Results; 

G.  Block  X,  Large-Group  Validation  Results;  and,  H.  Trainee  Background  Data. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  Number: 
Authoring  Aid  Type: 


Programmed  texts  and  audio-visual 
Self-paced,  individualized  instruction 
Formative  and  summative  evaluation 
Military  (Air  Force) 

1.4  Analyze  Existing  Course 
Procedure 
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Number  of  Pages:  62 

I Number  of  References:  None 

I 
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REFERENCE  NUMBER:  1.5.1 

SELECT  INSTRUCTIONAL  SETTING 

CITATION: 

Ely,  D.  P.  Instructional  hardware/A  guide  to  architectural  requirements. 

New  York:  Educational  Facilities  Laboratories,  1970.  (ERIC  Document 

Reproduction  Service  No.  ED  043  959) 

ABSTRACT: 

A dated  but  comprehensive  discussion  of  audio  and  visual  display  equipment, 
television-based  teaching  systems,  computer  assisted  instruction  (CAI),  and 
student  response  systems  useful  in  a variety  of  instructional  settings  is  presented. 
No  evaluation  of  specific  systems  is  given.  No  arguments  are  made  for  which 
systet.s  should  be  installed,  how  large  the  installations  should  be,  or  how  they 
might  be  used.  The  report  is  intended  to  "familiarize  the  educational  conmunity 
with  the  general  nature  of  various  electronic  teaching  systems."  Chapter  sections 
begin  with  a general  discussion,  and  then  specific  equipment  types  are  considered 
Approximate  costs  and  component  flowcharts  are  frequently  presented  for  system 
layouts. 

The  audio  equipment  chapter  considers:  (a)  tape  recorders  and  phonographs; 

(b)  wireless  teaching  systems  emphasizing  radio;  (c)  language  laboratories; 

(d)  telephone-based  teaching  systems;  (e)  dial  access  systems;  and  (f)  cable 
distribution  systems. 

Sections  in  the  visual  display  equipment  chapter  include:  (a)  slide  and 

filmstrip  projection;  (b)  overhead  projection;  (c)  opaque  projection;  (d)  motion 
picture  projection;  (e)  film  loops  including  single  concept,  repeating,  and 
cartridge  films;  (f)  microfilm,  (g)  room  darkening  and  ambient  light  standards; 

(h)  projection  screens;  (i)  lectern  control  panels;  (j)  individual  student 
carrels;  and  (j)  teaching  machines. 

In  the  chapter  on  television-based  teaching  systems,  instructional  television 
(ITV)  is  distinguished  from  educational  television  (ETV).  ITV  is  defined  as  tele- 
vision programs  planned  for  particular  school  systems  and  broadcast  via  closed 
circuit  transmission  to  specific  areas  or  classrooms.  ETV  is  defined  as  general 
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educational  and  Informational  television  programs  usually  broadcast  over  public 
television  channels.  The  chapter  Includes  sections  on:  (a)  television  transmission 

and  distribution  systems;  (b)  production  equipment;  (c)  production  space  require- 
ments; (d)  service  requirements  for  TV  production.  Including  air  conditioning,  power, 
and  lighting  requirements;  (e)  display  methods  for  TV  programs;  (f)  approximate 
equipment  costs  for  open  and  closed  circuit  systems;  and  (g)  Investment  cost 
(equipment  plus  maintenance)  for  150-classroom  and  300-classroom  systems. 

The  CAI  chapter  discusses:  (a)  Implications  of  educational  computer  systems; 

(b)  processing  requirements;  (c)  terminal  requirements;  (d)  system  options; 

(e)  space  requirements;  (f)  environmental  requirements;  (g)  service  requirements, 
including  methods  of  cabling  and  wiring  for  computer  systems;  (h)  fire  protection; 
and  (1)  approximate  costs  of  a large  municipal  CAI  system.  Due  to  constant 
development  of  computer-based  education  equipment,  costs  cited  by  the  author  are, 
perhaps,  now  unrealistic.  But  the  general  Information  In  each  section,  the  learning 
center  diagrams  of  typical  CAI  learning  centers,  and  the  wiring/cabling  Illustra- 
tions remain  Informative. 


Student  response  systems  use  some  type  of  responder  unit  at  a student  station 
and  an  Instructor  console  that  registers  responses.  The  chapter  contains  two 
sections:  (a)  a brief  description  of  responder  units  and  instructor  consoles; 

and  (b)  a discussion  of  three  conmercial  systems. 


Instructional  technology  terms  are  defined  In  Appendix  1,  a glossary  of  196 
definitions.  Other  appendices  provide  the  names  and  addresses  of  major  manu- 
facturers of  electronic  teaching  equipment,  a bibliography  of  suggested  readings, 
and  data  sheets  for  recording  equipment  characteristics. 


DATA  SUMMARY: 
Media: 

Application; 
Environment: 
Key  number: 


Audio-visual,  television,  computer-assisted 

instruction 

None 

None 

1.5  Select  Instructional  Setting 
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Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Methodology 

124+ 

15 
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REFERENCE  NUMBER:  2.1.1 

DEVELOP  OBJECTIVfS 

CITATION: 

Barton,  6.  E.  Performance  objectives:  A self- instructional  booklet.  Provo, 

Utah:  Brigham  Young  University  Press,  1973. 

ABSTRACT: 

This  booklet  teaches  how  to  recognize  and  write  technically  correct  performance 
objectives.  Validated  over  a two-year  period  by  hundreds  of  students  and  several 
instructional  psychologists,  the  booklet  is  arranged  in  programmed-text  format 
and  consists  of  two  parts. 

Part  One  covers  the  three  aspects  of  a technically- correct  performance 
objective:  behavior;  criteria;  and  conditions.  With  clear  examples.  Barton 

notes  that:  (a)  specified  behavior  should  be  observable  and  indicate  proficiency 

at  a task:  (b)  specified  criteria  should  be  measurable  and  appropriate,  i.e., 
should  match  or  be  important  qualifications  of  the  desired  behavior;  and 
(c)  specified  conditions  should  describe  the  exact  task  characteristics  to  be 
mastered  and  list  the  resource  materials  available  for  use. 

A self-check  is  given  at  the  end  of  each  section.  The  last  self-check  serves 
as  a post-test  for  the  program  and  can  be  taken  at  any  time. 

Part  Two  of  the  booklet  consists  of  pages  for  writing  technically  correct 
performance  objectives,  and  a key  for  critiquing  the  written  performance 
objectives. 

The  author  admits  that  writing  instructional  objectives  is  not  always 
appropriate  and  includes  an  article  discussing  situations  where  more  general 
learning  statements  — called  "worthwhile  activities"  — should  be  used. 

DATA  SUrtlARY: 

Media:  Programmed  text 

Application:  Self-paced  instruction 

Evaluation:  "Hundreds  of  students  and  several  instructional 

psychologists  over  a two-year  period" 
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Environment: 

Key  number: 
Authoring  and  type: 
Number  of  pages: 
Number  of  reference: 


Ci vi 1 i an 

2.1  Develop  Objectives 

Tool 

36 


None 
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REFERENCE  NUMBER:  2.1.2 

DEVELOP  OBJECTIVES 

CITATION: 

Behavioral  objectives  training  package.  Amherst,  Mass.:  Evaluation  Service 

Center  for  Occupational  Education,  1971.  (ERIC  Document  Reproduction  Service 

No.  ED  060  218) 

ABSTRACT: 

The  purposes  of  this  training  package  are:  (a)  to  serve  as  a user's  guide  to 

writing  and  implementing  effective  behavioral  objectives}  (b)  to  provide  an 
overview  of  the  services  provided  by  the  Evaluation  Service  Center  for  Occupational 
Education  (ESCOE);  and  (c)  to  suggest  an  alternative  technique  to  aid  the  writing  of 
behavioral  objectives. 

Part  I defines  behavioral  objectives  in  terms  of  measurable  performance, 
conditions  under  which  the  performance  occurs,  and  extent  to  which  the  performance 
can  be  measured.  These  terms  are  analogous  to  the  commonly- accepted  terms  of  the 
basic  structure  of  a behavioral  objective:  behavior;  condition;  and  standard. 

Part  II  discusses  the  operation  of  ESCOE.  Essentially,  the  service  is  an 
information  feedback/evaluation  system  for  a decentralized,  voluntary  network  of 
occupational  schools  called  l.ocal  Education  Agencies  (LEAS).  The  LEAS  submit  raw 
behavioral  objectives  to  ESCOE,  which  are  then  coded  and  placed  in  a computerized 
data  bank.  When  a sufficient  quantity  of  raw  (original)  objectives  are  available 
within  a given  subject  area,  they  are  retrieved  from  a data  bank  and  synthesized  by 
a panel  of  instructors  active  in  that  particular  subject.  The  objective  synthesis 
process  combines  raw  objectives  having  the  same  or  similar  performance  into  one 
larger  objective  so  that  all  variations  of  conditions  and  extent  indicated  by 
different  LEAs  are  included.  Criterion  tests  are  constructed  from  these  synthesized 
objectives  which  are  then  made  available  to  the  LEAs.  Participating  LEAs  can 
access  the  total  data  bank,  but  information  identifying  the  source  of  a given 
objective  is  blanked  out  when  a printout  is  sent  to  an  LEA  other  than  the  one 
submitting  the  original  data. 


AUTHORING  AID 
BIBLIOGRAPHY 


VOLUME  I 


MDC  El 570 
1 1 FEBRUARY  1977 


A description  of  each  section  and  Instructions  on  how  to  fill  In  the  required 
Information  are  given  for  the  objective  Input  coding  form  (Behavioral  Objective 
Reporting  Form  #12). 

Part  III  of  the  report  presents  an  article  entitled  "The  Operationalizing  of 
Fuzzy  Concepts,"  which  offers  an  alternative  way  of  systematically  breaking  down 
a generalization  Into  Its  measurable  parts,  I.e.,  Into  the  structure  suitable  for 
an  objective. 


Sample  transparencies 

are  available  from  ESCOE  for  participating  LEAs 

glossary  of  terms  and  phrases  Is  given. 

DATA  SUMMARY: 

Media: 

Computer-based,  print,  visual 

Application: 

Conventional  and  Individualized  Instruction 

Evaluation: 

None 

Environment: 

Civilian 

Key  number: 

2.1  Develop  Objectives 

Authoring  aid  type: 

Tool 

Number  of  pages: 

128 

Number  of  references: 

31 
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REFERENCE  NUMBER:  2.1.3 

DEVELOP  OBJECTIVES 

CITATION: 

Dillman,  F.  E.,  Jr.  Instructional  objectives:  Specificity  and  behavior. 

Menlo  Park,  California:  Dillman  Associates,  1971.  (ERIC  Document 

Reproduction  Source  No.  ED  061  704) 

ABSTRACT: 

The  basic  structure  of  a behavioral /performance  objective  contains  a required 
behavior,  a specified  standard,  and  a given  condition.  However,  there  are  good 
objectives  and  there  are  not-so-good  objectives.  To  write  good,  effective 
objectives,  Dillman  suggests  that  several  conceptual  factors  be  considered.  His 
discussion  is  presented  in  prog rammed- text  format  with  a self- review  and  answers 
and  explanations  at  the  end  of  each  section. 

The  first  factor  is  to  ensure  that  an  instructional  objective  rather  than  an 
educational  goal  is  written.  "An  educational  goal  is  a long-range  and  broader 
accomplishment  to  be  achieved  in  a semester,  in  a school  year,  or  even  during  a 
longer  school  experience,  such  as  upon  completion  of  high  school.  An  instructional 
objective  is,  by  contrast,  a short-range  accomplishment  to  be  achieved  in  a day, 
a week  - a relatively  shorter  time." 

The  second  factor  is  to  prevent  an  objective  from  being  written  so  narrowly 
and  restrictively  that  it  becomes  merely  a learning  activity  or  teaching  step. 

But  "if  you  are  writing  objectives  that  are  too  narrow,  it  doesn't  necessarily 
mean  that  you  should  'throw  th«n  out'  — when  you  get  your  objectives  properly 
composed,  the  'too  narrow'  objectives  may  still  be  used  as  teaching  points  or 
learning  activities  for  the  objectives  you  develop."  In  other  words,  objectives 
by  nature  have  a requisite  specificity. 

The  third  factor  is  to  write  both  the  content  and  the  language  of  the 
objective  to  the  level  of  the  target  population,  for  significant  behavior  at  one 
level  may  be  insignificant  at  another.  "Count  to  ten  in  one  minute  without 
external  aids"  may  be  a proper  objective  for  first-graders  but  not  for  calculus 
students  — and  yet  the  objective  contains  the  basic  parts:  a behavior,  a standard, 
and  a condition. 
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The  fourth  factor  is  to  not  bury  the  Intent  of  the  objective  in  the  preciseness 
of  its  words.  "The  problem  in  this  area  arises  when  a objective  states  that  the 
student  be  able  to  cut  out  the  pets  in  the  picture."  Does  this  require  skill  in 
cutting,  or  in  discriminating?  Does  it  call  for  indicator  behavior  or  principal 
performance?  If  you  want  the  learner  to  identify  the  pets,  then  demand  identifi- 
cation. It  is  unfair  to  demand  cutting  (and  the  connotation  to  the  pupil  that 
this  is  the  required  skill)  when  identification  is  what  the  objective  really 
requi res . 


Awareness  of  these  factors  plus  skill  in  writing  the  basic  parts  of  a behavioral 
objective  should  result  in  construction  of  good,  effective  objectives. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 
Authoring  and  type: 
Number  of  pages: 


Prograitmed  text 

Conventional  and  self-paced  instruction 

None 

Civilian 

2.1  Develop  Objectives 

Tool 

39 


Number  of  references:  None 
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REFERENCE  NUMBER:  2.1.4 

DEVELOP  OBJECTIVES 

CITATION: 

Objectives  for  Instructional  programs.  Reference  wail  chart.  Huntington  Beach, 

California:  INSGROUP  Inc.,  1975. 

ABSTRACT: 

Objectives  for  instructional  programs  is  a reusable,  multimedia  instructional 
package  providing  up  to  25  hours  of  group  or  self-paced  instruction  in  writing  and 
using  measurable  objectives.  The  Reference  Wall  Chart  abstracted  here  is  a four- 
color  wall  chart  containing  the  major  points  of  the  instructional  package  in  a 
concise  but  easily  read  format. 

The  chart  has  four  divisions.  The  first  division  identifies  the  structure 
of  measurable  objectives  as:  (a)  an  outcome,  which  is  also  known  as  "behavior;" 

(b)  an  achievement  level,  also  known  as  a "criterion;"  and  (c)  the  evaluation 
conditions,  also  known  as  simple  "conditions."  An  extensive,  categorized  "shopping 
list"  of  outcome  verbs  that  can  be  used  to  specify  achievement  levels  and  evalua- 
tion conditions  is  given. 

The  second  division  discusses  three  applications  of  measurable  objectives: 

(a)  learning  objectives;  (b)  organizational  objectives;  and  (c)  program  objectives. 
Definitions  and  examples  for  each  application  are  given. 

The  third  division  presents  several  taxonomies  of  learning  used  to  analyze  sets 
of  objectives  in  terms  of  logical  sequence  and  complexity  levels.  Eight  taxonomies 
in  the  cognitive,  affective,  and  psychomotor  domains  are  given.  A simplified, 
INSGROUP-derived,  seven-level  classification  scheme  for  sequencing  sets  of  objec- 
tives on  the  basis  of  complexity  is  illustrated. 

The  fourth  division  contains  brief  annotated  citations  for  taxonomies  presented 
on  the  chart,  and  gives  three  addresses  where  collections  of  objectives  may  be 
obtained. 
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DATA  SUFflARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Multicolor  visual 

Conventional  and  individualized  instruction 
None 

Civilian 

2.1  Develop  Objectives 
Tool 
1 

15 
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REFERENCE  NUMBER:  2.1.5 

CITATION: 

Preparation  of  learning  objectives.  Washington,  D.C.:  Department  of  the  Navy, 

Bureau  of  Naval  Personnel,  1968.  (ERIC  Document  Reproduction  Service  No. 

ED  058  721) 

ABSTRACT: 

Prepared  by  the  Bureau  of  Naval  Personnel,  this  programmed  workbook  is  intended 
to:  (a)  introduce  the  subject  of  learning  objectives  and  explain  their  use  and 

importance;  (b)  describe  a Navy  handbook  on  writing  learning  objectives  and  teaching 
students  how  to  use  the  handbook  as  a working  reference  guide;  and  (c)  provide 
practice  in  preparing  relevant,  practical  learning  objectives.  Intended  for  military 
personnel,  the  workbook  is  useful  for  civilian  practitioners  too.  The  workbook 
contains  three  modules. 

In  Module  I,  Chapter  1 defines  a learning  objective  as  "an  instructional  goal 
expressed  in  terms  of  measurable  student  performance."  Chapter  2 defines  terminal 
objectives  as  representing  what  the  student  will  be  doing  on  the  job  while  enabling 
objectives  should  close  the  gap  between  what  the  student  knows  now  and  what  he  is 
to  learn.  It  is  noted  that  an  objective  is  classified  as  either  terminal  or  enabling 
according  to  its  purpose  within  the  curriculum,  and  that  what  represents  terminal 
performance  for  one  course  may  represent  enabling  performance  for  a later,  more 
advanced  course  or  skill.  Chapter  3 states  that  learning  objectives  are  important 
because  they  support  "the  overall  system  of  Consistent  Course  Design,  Practical 
Instructor  Guidance,  and  Job-Effective  Student  Instruction." 

Module  II  introduces  the  Handbook  for  Writing  Learning  Objectives  (NAVPERS 
93510-2),  the  guidebook  of  the  Bureau  of  Naval  Personnel  for  developing  learning 
objectives.  Module  II  is  written  in  a SCRIPT  format  (^elt-Confirming  Recitation 
in  Rrogrammed  Teaching)  and  presents  a series  of  questions  about  the  text.  The 
format  permits  a student  to  pass  quickly  over  known  material,  but  directs  him  to 
sections  of  the  Handbook  needed  for  study.  Upon  completion  of  Module  II,  a review 
quiz  is  given  that  requires  a student  to:  (a)  select  the  scope  of  course,  topic, 

and  lesson  objectives;  (b)  identify  each  of  the  three  functional  elements  of  a 
learning  objective,  without  error,  when  presented  with  a model  objective; 
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(c)  select  examples  of  knowledge,  skill,  and  attitude  categories  of  learning; 

(d)  identify  two  purposes  of  a Topic  Analysis  Worksheet,  when  presented  alternative 
answers  in  a multiple-choice  format;  and  (e)  identify  three  steps  in  learning 
objective  analysis,  when  presented  with  alternative  answers  in  a multiple-choice 
format. 

Module  III  contains  an  exercise  kit  for  using  the  Navy  handbook  as  well  as 
the  handbook  itself.  Containing  matching  sets,  constructed-response  items,  and 
similar  learning-by-doing  exercises.  Module  III  covers  the  following  topics: 

(a)  concept-matching;  (b)  evaluating  learning  objectives;  (c)  constructing  learn- 
ing objectives;  and  (d)  a careful ly- structured  final  test  in  which  the  student 
writes  a knowledge,  attitude,  and  a skill  objective.  The  test  is  graded  upon 
completion.  The  scoring  system  consists  of  stars.  For  each  question  answered 
"Yes,"  2 stars  are  awarded;  for  each  question  answered  "No,"  no  star  is  awarded. 

A rating  scale  ranges  from  6 stars  or  less  to  10  stars.  Ten  stars  represents 
"outstanding"  while  6 stars  or  less  is  "unsatisfactory."  The  scoring  system  could 
be  upgraded. 

DATA  SUMMARY: 


Media: 

Programmed  text 

Application: 

Self-paced  instruction 

Evaluation: 

None 

Envi ronment: 

Military  (Navy) 

Key  number: 

2.1  Develop  Objectives 

Authoring  aid  type: 

Tool 

Number  of  pages: 

172 

Number  of  references: 

None 
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REFERENCE  NUMBER:  2.1.6 

DEVELOP  OBJECTIVES 

CITATIONS: 


Rose,  B.  K. , Balasek,  J.,  Kelleher,  J.,  Lutz,  J.  L.,  & Nelken,  I.  A progratmied 
course  for  the  writing  of  performance  objectives.  A constructed  response 
linear  program.  Bloomington,  Indiana:  Phi  Delta  Kappa,  Comnission  on 

Educational  Planning,  1972.  (ERIC  Document  Reproduction  Service  No.  ED 
073  528) 

Rose,  B.  K.,  Balasek,  J.,  Kelleher,  J.,  Lutz,  J.  L.,  & Nelken,  I.  Writing 

performance  objectives.  Instructor's  manual  for  teachers  and  administrators. 

To  be  used  with  a programmed  course  for  writing  of  performance  objectives. 
Bloomington,  Indiana:  Phi  Delta  Kappa,  Commission  on  Educational  Planning 

1972.  (ERIC  Document  Reproduction  Service  No.  ED  073  529) 

ABSTRACT: 

Designed  to  help  teachers  learn  how  to  write  performance  objectives,  this  well- 
written  programmed  text  covers:  (a)  identifying  goal  statements;  (b)  identifying 

basic  parts  of  a performance  objective;  (c)  writing  performance  objectives; 

(d)  writing  cognitive  objectives  at  the  levels  of  knowledge,  comprehension,  appli- 
cation, analysis,  synthesis,  and  evaluation;  (e)  recognizing  psychomotor  objectives 
at  the  levels  of  imitation,  manipulation,  precision,  articulation,  and  naturaliza- 
tion (that  is,  achieving  automatic  and  spontaneous  responses);  (f)  recognizing  and 
writing  performance  objectives  in  the  affective  domain;  and  (g)  recognizing  and 
writing  performance  objectives  at  the  program  level  and  the  instruction?  level. 

Each  part  of  the  text  begins  with  objectives  for  the  part  and  continues  with 
logically-arranged  frames  containing  frequent  embedded  questions  and  feed  back. 

Part  II  teaches  how  to  classify  performance  objectives  into  the  cognitive,  psycho- 
motor, and  affective  domains,  and  how  to  specify  which  level  in  a domain  the 
objective  is  intended  to  measure.  Part  III  teaches  how  to  write  program  level 
objectives  relating  to  institutional  goals  and  reflecting  desired  exit  levels  of 
student  performance.  Part  III  teaches  how  to  write  instructional  level  objectives 
for  specific  courses,  disciplines,  or  units  of  work  by  grade  level  or  curriculum 
area. 
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A series  of  assignments  intended  to  review  the  previous  information  concludes 
the  book.  Topics  covered  include  recognizing  complete  and  incomplete  performance 
objectives,  completing  performance  objectives  in  two  of  six  suggested  curriculum 
areas,  and  identifying  simple  and  complex  performance  objectives. 

The  instructor's  manual  accompanying  the  programmed  text  is  designed  to  enable 
school  districts  to  utilize  effective  management  procedures  for  inservice  training 
of  teachers  in  writing  performance  objectives.  The  first  four  sections  of  the 
instructor's  manual  provide  management  guidelines  and  instructional  aids  for 
defining  and  reviewing  the  use  of  programmed  materials.  Management  guidelines 
for  developing  programmed  materials  are  given  in  Section  5.  The  presentation 
includes  a description  of  sequential  learning  paths  as  an  organizational  technique 
for  instructional  system  design. 

DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Key  number: 

Authoring  aid  type: 

Number  of  pages: 

Number  of  references: 


Programmed  text  (Course),  Print  (Handbook) 
Self-paced  instruction  (Course  and  Handbook) 
None  (Course  and  Handbook) 

2.1  Develop  Objectives  (Course  and  Handbook) 
Tool  (Course  and  Handbook) 

49  (Course),  38  (Handbook) 

17  (Course),  None  (Handbook) 
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REFERENCE  NUMBER:  2.2.1  DEVELOP  TESTS 

CITATION: 

Brown,  L.,  Brooks,  G.,  Cocks,  P.  J.,  and  Kersch,  M.  E.  Evaluation  for  Individualized 
instruction.  An  operational  guide  for  teacher  workshops.  Downers  Grove,  111.: 
Institute  for  Educational  Research,  1972.  (ERIC  Document  Reproduction  Service 
No.  ED  066  492) 

ABSTRACT: 

Presented  in  this  operational  guide  for  teacher  workshops  are  11  instructional 
sessions  concerned  with:  (a)  educational  objectives;  (b)  global  objectives; 

(c)  behavioral  objectives;  (d)  hierarchy  of  learning  objectives;  (e)  purposes  of 
achievement  testing;  (f)  techniques  of  measurement;  (g)  multiple-choice  items; 

(h)  basic  statistical  measures;  (i)  test  analysis  interpretation  of  computer 
output;  (j)  validity  and  reliability,  and  (k)  the  liaison  role  of  the  workshop 
participant.  The  discussion  relating  to  testing  is  given  below. 

Multiple-choice  test  items  consist  of  two  parts,  a stem  or  lead  and  alternatives. 
Faults  can  be  written  unintentionally  into  either  part.  A list  of  10  common  faults 
of  multiple-choice  test  construction  is  followed  by  a fault  exercise  sheet.  The 
faults  are:  (a)  behavior  elicited  by  test  items  does  not  match  the  behavior  stated 

by  the  applicable  objective;  (b)  the  item  stem  does  not  present  a single,  definite, 
important  problem;  (c)  the  item  alternatives  do  not  include  a single,  correct,  best 
answer  or  completion;  (d)  an  incomplete  item  stem  does  not  include  as  much  of  the 
problem  as  possible,  which  may  lead  to  several  possible  correct  alternatives; 

(e)  the  item  alternatives  are  not  stated  as  briefly  as  possible;  (f)  the  item  is 
not  stated  in  positive  terms  and,  when  a negative  is  used,  it  is  seldom  underlined 
or  capitalized;  (g)  the  item  alternatives  are  not  grammatically  consistent  with  the 
stem;  (h)  alternatives  are  not  approximately  similar  in  grammatical  construction, 
degree  of  precision,  or  length;  (i)  alternatives  are  implausible  and,  therefore, 
do  not  tempt  students  with  incomplete  or  superficial  knowledge  of  the  subject; 
and  (j)  alternatives  contain  a "trick"  word  in  an  otherwise  correct  response  just 
for  the  purpose  of  catching  the  reader  offguard. 
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In  the  test  analysis  Instructional  session,  brief  discussions  of  item  dif- 
ficulty, item  discriminating  power,  and  correct  scores  are  presented,  A series 
of  computer  printouts  of  various  test  analyses  follow  the  discussion,  and  then  a 
list  of  52  study  questions  in  four  categories  is  given:  (a)  format  and  general 

data;  (b)  test  items;  (c)  students  as  individuals  and  as  groups;  and  (d)  uses  of 
data  for  decision-making. 

Creation  of  a test-item  data  bank  was  one  purpose  of  the  workshops.  It  is 
unknown  at  this  time  whether  a data  oank  was  in  fact  created. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Computer-based 

Conventional  and  individualized  instruction 
None 

Civilian 

2.2  Develop  Tests 
Tool 
90 
9 
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REFERENCE  NUMBER:  2.2.2 

DEVELOP  TESTS 

CITATION: 

CSE-RBS  test  evaluations:  Tests  of  higher-order  cognitive,  affective,  and 

interpersonal  skills.  Los  Angeles:  University  of  California,  Graduate 

School  of  Education,  Center  for  the  Study  of  Evaluation,  1972. 

ABSTRACT: 

The  Center  for  the  Study  of  Evaluation  (CSE)  and  Research  for  Better  Schools 
(RBS),  Inc.  conducted  a study  to:  (a)  determine  the  number  and  quality  of  testing 
instruments  available  for  measuring  higher-order  cognitive,  affective,  and  inter- 
personal skills;  and  (b)  to  reveal  areas  where  such  instruments  were  not  available. 
Approximately  2600  tests  and  scales  were  collected  and  evaluated.  Of  429  categories 
in  the  three  areas,  183  (43%)  had  no  available  test,  and  179  (42%)  contained  10 
or  fewer  tests.  The  quality  of  the  instruments  was  rated  predominately  poor  to 
fair. 

An  evaluation  instrument  was  developed  which  rated  all  instruments  according 
to  seven  criteria:  (1)  Validity;  (2)  Examine  appropriateness,  i.e.,  its  suitability 

for  an  examiner  or  examinee  in  terms  of  comprehension  level,  physical  format  and 
response  mode;  (3)  Normed  excellence  concerning  reliability,  replicability,  and 
refinement  of  test  results;  (4)  Teaching  feedback;  (5)  Usability}  (6)  Retest 
potential,  meaning  the  number  of  alternate  testing  forms  available;  and  (7)  Ethical 
propriety,  which  considered  questionable  administrative  procedures  or  test  content, 
and  treatments  which  were  inappropriate,  insulting,  or  offensive  to  the  examinee. 

As  the  underlined  letters  indicate,  the  evaluation  instrument  was  called  VENTURE. 

The  section  for  tests  of  higher-order  cognitive  skills  included  measures  of 
disposition,  cognitive  styles,  strategies,  and  skills.  A two-dimensional  class- 
ification scheme  was  developed  for  the  evaluation  of  cognitive  tests.  Rows  were 
labeled  with  category  descriptors  such  as  "classifying"  and  "inventing  - ideating." 
Columns  were  headed  by  content  identifiers  such  as  "verbal -semantic,"  and 
"symbolic."  Cells  indicated  the  code  assigned  to  the  interaction  between  the  two 
dimensions,  and  included  the  number  of  tests  collected  and  analyzed  in  each  category. 
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The  section  for  tests  of  affective  skills  generally  included  measures  of 
traits  or  dispositions  that  hold  constant  for  an  individual  in  different  situations. 
As  in  the  previous  section,  a two-dimensional  matrix  for  test  evaluation  was 
developed.  Rows  were  identified  by  six  affective  skills  categories  which  in  some 
cases  were  subcategorized.  Six  major  affective  skills  were  listed:  (1)  personal 

temperament;  (2)  social  temperament;  (3)  attitudes,  opinions,  beliefs;  (4)  needs; 

(5)  interests;  and  (6)  values.  Columns  were  labeled  with  classification 
descriptors  such  as  "Reports  About  Self:  Description"  and  "Behavioral  Measures: 
Actual  Measures."  Cells  contained  assigned  codes  and  the  number  of  tests  collected 
and  evaluated  for  each  category  or  subcategory. 

In  the  third  section,  tests  of  interpersonal  skills  focusing  on  specific, 
concrete,  social  situations  and  measuring  the  behaviors  exhibited  by  an  individual 
in  those  situations  were  evaluated.  This  aspect  generally  distinguishes  tests  of 
interpersonal  skills  from  tests  of  affective  skills.  For  example,  a test  which 
measured  aspects  of  a student's  relationship  with  his  actual  teachers  was  assigned 
to  the  interpersonal  skills  section;  but  a test  which  measured  the  valence  of  a 
student's  attitude  toward  teachers  in  general  was  listed  in  the  section  on  affec- 
tive skills.  A three-dimensional  morphological  model  evaluated  test  type  by  role 
by  situation.  Test  types  were:  (a)  reports  about  self;  (b)  reports  about  others; 

and  (c)  behavioral  measures.  Roles  included  relations  with:  (1)  peers; 

(2)  authorities;  (3)  subordinates;  and  (4)  people- in- general.  Situations  for  the 
first  roles  included:  (a)  at  school;  (b)  at  home;  (c)  other;  and  (d)  general. 

For  the  last  role,  the  situation  "at  work"  replaced  "other"  and  "general."  Cells 
contained  assigned  codes  for  each  test  category  and  the  number  of  tests  collected 
and  evaluated  in  that  category. 

Two  indexes  included  in  the  volume  provide:  (a)  the  tests  and  scales  arranged 

in  alphabetical  order  with  section/cateogry  number;  and  (b)  publication,  distri- 
' bution,  and  reference  sources  for  the  tests  and  scales. 
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REFERENCE  NUMBER:  2.2.3 

DEVELOP  TESTS 

CITATION: 

Gronlund,  N.  Preparing  criterion-referenced  tests  for  classroom  instruction. 

New  York:  Macmillan,  1973, 

ABSTRACT: 

Gronlund  presents  a practical  guide  to  preparing  and  using  criterion-referenced 
tests  (CRT)  in  classroom  instruction.  Chapter  1 discusses  the  nature  of  six 
requirements  for  effective  CRT.  The  six  requirements  are:  (a)  a clearly  defined 

and  specified  domain  of  learning  tasks;  (b)  clearly  specified  standards  of  per- 
formance; (c)  clearly  defined  behavioral  objectives;  (d)  adequate  sampling  of 
student  performance  within  each  area  of  performance;  (e)  test  items  that  reflect 
the  behavior  specified  in  the  instructional  objectives;  and  (f)  criterion  scores 
measuring  performance  to  an  objective  standard. 

Chapters  2 and  3 discuss  use  of  CRT  at  the  mastery  level  (i.e.,  achievement  of 
minimum  standards)  and  at  the  developmental  level  (i.e.,  achievement  beyond  minimum 
standards,  such  as  thinking  and  understanding  skills). 

Chapter  4 describes  the  procedures  for  planning  a test  as:  (a)  delimit  the 

unit  of  instruction  to  be  measured;  (b)  state  the  objectives  in  specific  terms; 

(c)  make  a content  outline;  (d)  prepare  a table  of  specifications;  (e)  set  pass- 
fail  standards;  (f)  select  the  type  of  test  item  to  be  used;  (g)  write  the  test 
items;  (h)  organize  the  item  pool  into  a test;  and  (i)  create  a test-results 
report  form  for  student  use. 

Chapter  5 tells  how  to  write  short-answer,  multiple-choice,  true-false,  and 
matching  test  items.  It  also  includes  several  rules  for  writing  test  items.  They 
art:  (1)  ensure  that  the  item  elicits  the  specific  behavior  called  for  in  the 

objective;  (2)  make  the  task  to  be  performed  clear  and  definite;  (3)  delete  non- 
functional material  from  the  item;  (4)  reject  irrelevant  factors  that  prevent  the 
student  from  correct  responses;  (5)  delete  irrelevant  clues  that  might  lead  to 
unintended  correct  or  incorrect  responses;  (6)  write  the  test  item  at  the  proper 
level  of  difficulty;  (7)  insure  that  the  correct  answer  is  agreed  upon  by  subject- 
matter  experts;  (8)  write  the  item  in  positive  form  whenever  possible;  and  (9)  write 
enough  test  items  so  that  a validated  test  item  pool  may  be  selected. 

B-50 


MCooMMmLL  DOuoLJkm  MmrmoMJkUTtcm  coa«m«/wv-  mjtmT 


AUTHORING  AID 
BIBLIOGRAPHY 


VOLUME  I 


MDC  El 570 
n FEBRUARY  1977 


Chapter  6 discusses  the  use  and  appraisal  of  criterion-referenced  tests. 

Three  examples  of  tables  of  specifications  for  40- , 60-,  and  100-item  tests 
are  given  which  allow  a clear  breakout  between  the  number  of  test  items  per  subject 
content  area  and  specific  performance  type.  Appendix  A contains  a check  list  for 
evaluating  CRTs  which  may  be  used  after  test  item  construction  to  ensure  that  the 
test  plan,  test  items,  and  test  format  will  adequately  measure  behavioral  outcomes. 

DATA  SUMMARY: 


Media: 

Print 

Application: 

Conventional  and  individualized  instruction 

Evaluation: 

None 

Environment: 

Civilian 

Key  number: 

2.2  Develop  Tests 

Authoring  aid  type: 

Tool 

Number  of  pages: 

55 

Number  of  references: 

12 

B-51 


tMcoOMMmLL  oOLiat^m  MmTmMtuumcm  coavkmwv’  - mMmr 


AUTHORING  AID 
BIBLIOGRAPHY 

VOLUME  1 

MDC  El 570 
11  FEBRUARY  1977 

REFERENCE  NUMBER:  2.2.4 

DEVELOP  TESTS 

CITATION: 

Stevens,  J.  C.,  and  O'Neil,  H.  F.,  Jr.  Some  guidelines  for  development  and  review 
of  multiple-choice  questions.  Austin:  University  of  Texas  Press,  1976. 

ABSTRACT: 

This  report  from  Stevens  and  O'Neil  is  divided  into  two  sections.  The  first 
section  is  a discussion  of  the  advantages  of  multiple-choice  questions,  the 
different  forms  of  multi pie- choice  questions,  and  methods  for  varying  item  * 
difficulty.  The  second  section  lists,  with  examples,  eight  guidelines  for  writing 
and  reviewing  multiple-choice  questions.  The  guidelines  are  as  follows.  First, 
the  multiple-choice  item  should  be  meaningful  by  itself  and  should  present  a 
definite  problem.  Second,  the  item  stem  should  include  as  much  of  the  problem 
as  possible  and  should  be  free  from  irrelevant  material.  Third,  negatively-stated 
item  stems  should  be  avoided  whenever  possible.  Fourth,  all  alternatives  should 
be  grammatically  consistent  with  the  stem  of  the  item.  Fifth,  a multiple-choice 
item  should  contain  only  one  correct  or  clearly  best  answer.  Sixth,  all  alterna- 
tives should  be  plausible;  ensure  this  by  drawing  alternatives  from  the  same  domain 
of  information,  by  excluding  uncorrelated  facts,  and  by  considering  the  previous 
learning  experience  of  the  student.  Seventh,  avoid  verbal  association  between  the 
stem  and  the  correct  answer;  an  example  of  verbal  association  between  a word  in 
the  stem  (locality)  and  a word  in  the  alternative  (local)  is  given  by  the  follow- 
ing question: 

"Which  of  the  following  agencies  should  you  contact  to 
find  out  about  a tornado  warning  in  you  locality"? 

a.  State  farm  bureau 

b.  Local  radio  station 

c.  U.S.  Post  Office 

d.  U.S.  Weather  Bureau. 

Eighth,  the  relative  length  of  the  alternatives  should  not  provide  a clue  to  the 
answer. 
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Three  additional  guidelines  are  given  without  examples:  (a)  seldom  use 

special  alternatives  such  as  "none  of  the  above"  or  "all  of  the  above";  (b)  place 
the  correct  answer  in  each  alternative  position  approximately  an  equal  number  of 
times,  but  in  random  order;  and  (c)  have  a competent  colleague  independently  review 
the  test  items. 

In  addition,  twelve  undesirable  multiple-choice  item  constructions  with  examples 
and  four  desirable  constructions  with  examples  are  given. 
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REFERENCE  NUMBER:  2.2.5 
CITATION: 

Stevens,  J.  C.,  and  O'Neil,  H.  F.,  Jr.  Suggestions  for  development  of  test  items. 

Austin:  University  of  Texas  Press,  1976. 

ABSTRACT: 

An  operational  definition  for  test  item  development  has  two  components.  First, 
test  item  development  is  a set  of  operations  by  which  a syntactic  structure  is 
assigned  to  an  instructional  sentence.  Second,  test  item  development  is  a set  of 
operations  performed  on  that  syntax  which  transforms  the  relevant  segments  of  the 
instructional  sentence  into  test  items.  Stevens  and  O'Neil  argue  that  the  opera- 
tional definition  of  test  item  development  is  useful  for  deriving  test  items  from 
actual  instructional  sentences  through  a technique  called  item  transformation.  The 
technique  assigns  a syntactic  structure  to  an  instructional  sentence  in  terms  of 
subject,  verb,  object,  and  prepositional  phases  and  transforms  the  sentence  compo- 
nents into  questions.  The  technique  is  illustrated  with  five  different  types  of 
questions.  The  first  two  techniques  are  suitable  for  rote  memory  testing  situations 
the  last  three  are  intended  for  comprehension  and  higher  level  testing. 

First,  verbatim  instruction  can  be  made  into  yes/no,  true/false,  short  answer, 
and  multiple-choice  questions.  Yes/no  questions  can  be  derived  by  simply  taking 
a sentence  directly  from  the  instructional  material,  deleting  the  period,  and 
adding  a question  mark.  True/false  questions  can  be  made  by  simply  taking  a 
sentence  from  the  instructional  material  and  adding  the  appropriate  directions  at 
the  beginning  of  the  test;  since  most  true/false  tests  have  a large  porportion 
of  true  statements,  three  ways  are  suggested  for  deriving  false  items.  Short- 
answer  questions  can  be  created  by  identifying  the  desired  sentence  component 
(a  subject  phrase,  for  example)  and  deleting  it  for  the  question.  Multiple-choice 
questions  can  be  derived  by  a similar  procedure.  Base  sentences  are  transformed 
into  stems  by  identifying  the  segment  to  be  deleted  and  then  deleting  it. 
Alternatives  are  generated  by  identifying  the  structure  of  the  correct  response 
and  taking  words,  phrases,  or  sentences  with  similar  structure  from  the  instruc- 
tional materials  and  using  them  as  distractors. 


B-54 


AUTHORING  AID 

bibliography 


VOLUME  I 


MDC  El 570 
1 1 FEBRUARY  1977 


Second,  transformed  verbatim  instruction  can  be  made  into  questions  by  using 
the  wh- transformation.  To  accomplish  the  wh- transformation,  a test  writer  must: 

(a)  study  the  structure  of  the  sentence;  (b)  select  the  segment  to  be  tested; 

(c)  delete  the  segment;  (d)  replace  the  deleted  phrase  with  a wh-word  and 
(e)  shift  the  wh-word  to  the  front  of  the  sentence.  There  are  four  groups  of 
wh-words.  First,  who  and  what  are  used  to  replace  human  and  nonhuman  nouns, 
respectively.  Second,  which  is  used  to  replace  noun  modifiers.  Third,  how,  when, 
where,  and  are  used  to  replace  the  various  adverbial  categories.  Fourth,  a 
form  of  what  do  is  used  when  the  verb  is  deleted.  Alternatives  are  generated  by 
the  process  noted  above. 

Third,  paraphrase  questions  can  be  developed  by  paraphrasing  the  base  sentence, 
deleting  a segment  from  the  paraphrase,  or  transforming  the  paraphrase  into  a 
wh-question.  A statement  is  defined  as  a paraphrase  of  another  if  (a)  they  have 
no  substantive  word  in  common,  i.e.  nouns,  verbs,  and  modifiers,  and  (b)  they 
are  equivalent  in  meaning.  The  authors  note  that  not  all  sentences  can  be 
paraphrased,  especially  those  using  proper  nouns  and  words  of  a highly  specific 
or  technical  nature. 

Fourth,  questions  can  be  formed  by  substituting  particular  terms  for  super- 
ordinate terms.  A term  is  superordinate  if  at  least  some  of  the  members  of  the 
class  can  be  named  without  repeating  the  superordinate  term.  For  example,  the 
word  "tool"  is  a superordinate  term  because  each  member  of  the  class  has  a particu- 
lar name,  i.e.  pliers,  hanmers,  wrenches,  etc.  To  derive  a question  formed  by 
substituting  particular  terms  for  superordinate  terms,  (a)  replace  every  super- 
ordinate term  in  the  base  sentence  with  a particular  item,  (b)  substitute 
synonyms  for  every  remaining  substantive  word,  (c)  delete  a segment,  or  (d)  form 
a wh-question. 
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Fifth,  questions  can  be  formed  by  applying  the  opposite  process,  that  is, 
by  substituting  superordinate  terms  for  specific  terms.  The  meaning  of  the 
sentence  is  enlarged  but  not  changed  completely  by  the  procedure.  For  example, 
the  superordinate  term  "supplies"  was  substituted  for  the  particular  term  "pencils" 
in  the  following  sentences: 

base  sentence:  "The  phrase  Base  Service  Store  Item  tells  you  that  pencils 
are  not  subject  to  normal  issue  procedures." 

transformed  sentence:  "The  phrase  Base  Service  Store  Item  tells  you  that 

certain  supplies  are  not  subject  to  normal  issue 
procedures." 

To  create  questions  by  substituting  superordinate  terms  for  particular  terms, 

(a)  replace  particular  terms  with  superordinate  terms,  (b)  replace  all  substantive 
words  with  synonyms,  or  paraphrase  where  appropriate,  (c)  delete  a segment,  or 
(d)  form  a wh-question. 

Although  some  of  these  transformations  may  appear  quite  complex,  numerous 
examples  are  clearly  presented  which  show  how  each  transformation  "works"  on 
actual  instructional  materials. 
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REFERENCED  NUMBER;  2.2.6 

DEVELOP  TESTS 

CITATION: 

Swezey,  R.  W.,  and  Perlstein,  R.  B.  Developing  criterion-referenced  tests 
(Report  No.  287-AR18(2)-IR-0974-RWS) . Arlington,  Va.:  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  Unit  Training  and  Educational 
Technology  Systems  Area,  Performance  Measurements  and  Standards  Work  Unit, 

March  1974.  (NTIS  Document  Reproduction  Service  No.  AD  A014  987) 

ABSTRACT: 

Though  the  difference  between  normative-referenced  tests  and  criterion- 
referenced  tests  (CRT)  may  not  always  be  obvious,  Swezey  and  Pearl  stein  argue  that 
a test  is  criterion- referenced  if:  (a)  the  test  items  are  based  upon  training 

objectives  developed  from  performance  objectives  external  to  training;  (b)  the 

I 

test  itself  is  primarily  used  to  measure  mastery;  and  (c)  the  test  score  is 
based  upon  absolute  standards  such  as  job  competence  rather  than  relative  standards 
such  as  class  standings.  Based  on  those  three  criteria,  the  authors  present  a 
how-to  guide  for  development  and  administration  of  CRT.  The  guide  contains  review 
questions  and  answers  for  each  chapter  as  well  as  large  fold-out  flow  charts  which 
identify  the  CRT  developmental  step  to  which  a chapter  refers.  The  CRT  developmental 
process  is  divided  into  four  steps. 

The  first  step  is  to  develop  adequate  objectives.  Three  levels  of  objectives 
are  defined.  Level  1 objectives  are  prepared  according  to  doctrine  and/or 
actual,  meaningful  units  of  work  which  occur  in  an  operational  environment. 

Level  2 objectives  are  modified  Level  1 objectives  which  match  training  resources 
and  safety  requirements  and  indicate  what  a student  must  be  able  to  do  at  the 
end  of  training.  Level  3 objectives  — commonly  called  Enabling  Objectives  or 
Learning  Elements  — indicate  what  a student  must  know  and  do  as  prerequisites 
to  attaining  Level  2 objectives. 

The  second  step  is  to  develop  a test  plan  by  examining  practical  constraints, 
deciding  the  type,  number,  and  fidelity  (realism)  of  the  test  items,  planning 
item  sampling  or  sampling  among  conditions,  and  documenting  the  entire  test 
plan.  Practical  examples  and  a test  plan  worksheet  are  given. 
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The  third  and  fourth  steps  are  to  build  an  Item  pool  and  select  the  final  test 
Items  for  Instruction.  For  the  Item  pool,  the  authors  recommend  that  twice  as  many 
Items  be  developed  as  are  specified  In  the  test  plan  so  the  pool  can  be  evaluated, 
appropriate  Items  selected,  and  alternate  test  forms  developed. 

For  selecting  final  test  Items,  specific  guidelines  for  determining  the 
correct  tryout  sample  size  (versus  total  student  flow),  conducting  the  tryout, 
analyzing  tryout  results,  and  reducing  the  Item  pool  are  given.  The  PHI  coefficient 
was  chosen  as  an  Item  analysis  technique,  and  practical  examples  thoroughly 
Illustrate  Its  use.  Two  worksheets  are  given,  one  for  recording  feedback  from 
tryouts  and  another  for  summarizing  the  results  of  Item  pool  reviews  In  terms  of 
organizational  flow. 

The  final  sections  of  the  manual.  Administering  and  Scoring  CRTs  and  Assessing 
Reliability  and  Validity,  are  essentially  dependent  upon  specific  testing  environ- 
ments and  available  statistical  resources  and,  therefore,  may  not  be  readily 
transferable  as  authoring  tools  or  procedures. 

Five  appendices  are  Included:  (a)  Checklist  for  Constructing  CRTs;  (b)  Checklist 
for  Evaluating  CRTs;  (c)  Glossary;  (d)  Square  Root  Tables;  and  (e)  Review  Questions 
and  Answers. 
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REFERENCE  NUMBER  2.3.1 
CITATION: 

DESCRIBE  ENTRY  BEHAVIOR 

Bierbautn,  G.  A.,  and  Planisek,  R.  J.  An  index  and  procedure  for  readmitting  the 
the  academically  dismissed  student.  Kent,  Ohio:  Kent  State  University, 
1969.  (ERIC  Document  Reproduction  Service  No.  ED  063  555) 


ABSTRACT: 

An  index  and  procedure  were  developed  to  establish  the  probable  academic  success 
of  students  seeking  readmission  after  academic  dismissal.  Covering  the  period  1959 
to  1969,  data  were  obtained  on  1460  students  from  four  college  deans.  The  data 
were  analyzed,  and  an  index  and  procedure  based  on  the  minimum  grade  point  average 
(6PA)  needed  to  graduate  were  created. 

The  data  indicated  the  range  of  dismissals  for  students  returning  for  one  term 
as  follows:  (a)  for  freshmen,  the  range  of  dismissals  was  79%  to  94%;  (b)  for 

sophomores,  62%  to  79%;  (c)  for  juniors,  44%  to  56%;  and  (d)  for  seniors,  9%  to 
40%.  The  farther  along  a student  was  at  the  time  of  his  academic  dismissal,  the 
better  were  his  chances  for  graduating  after  readmission.  Readmission  data  indi- 
cated: (a)  10%  to  20%  finally  graduated;  (b)  35%  to  50%  were  dismissed  again;  and 

(c)  15%  to  30%  did  not  return  after  readmission. 

A chart  was  constructed  with  credit  hours  ranging  from  1 to  200  along  one 
dimension,  and  the  6PA  ranging  from  0 to  2.00  in  .05  increments  along  the  other 
dimension.  Two  indices  were  then  retrieved  from  the  cells  of  the  chart.  The  first 
index  computed  the  minimum  required  GPA  needed  each  term  after  readmission  for 
graduation.  The  GPA  multiplied  by  the  credit  hours  needed  to  graduate  (the  total 
was  called  'earned  quality  points')  minus  the  earned  quality  points  at  the  time  of 
dismissal  provided  an  indication  of  the  quality  point  deficit  to  be  overcome  for 
graduation.  The  second  index  calculated  the  difference  between  the  cumulative  GPA 
at  dismissal  and  the  minimum  required  GPA  needed  to  graduate.  Thus,  the  minimum 
GPA  to  be  maintained  each  semester  after  readmission  was  obtained. 

Given  the  two  indexes  and  the  four  samples  of  readmitted  students,  another 
chart  was  plotted  to  establish  a criticality  index.  Successful  readmissions  were 
plotted  as  stars,  and  unsuccessful  readmissions  were  plotted  as  black  dots.  A line 
drawn  to  optimize  the  discrimination  between  the  stars  and  the  dots  indicated  that, 
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for  the  Arts  and  Science  College,  a minimum  GPA  performance  of  2.20  was  required 
for  readmission. 

It  Is  noted  that  more  rigorous  statistical  approaches  could  be  considered  In 
relation  to  their  effectiveness  when  compared  to  the  ease  of  using  this  empirically- 
derived  Index.  Samples  plotted  over  time  may  suggest  trends  such  as  effects 
attributed  to  changing  academic  deans. 
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REFERENCE  NUMBER:  2.3.2 

DESCRIBE  ENTRY  BEHAVIOR 

CITATION: 

Ecternacht,  6.  J.,  Reilly,  R.  R.,  and  McCaffrey,  P.  J.  Development  and  validity 

of  a vocational  and  occupational  interest  inventory  (Report  No.  AFHRL-TR-73-38). 

Lackland  Air  Force  Base  Texas:  Air  Force  Human  Resources  Laboratory,  Personnel 

Research  Division,  December  1973, 

ABSTRACT: 

An  interest  inventory  designed  to  measure  the  vocational  interests  of  enlisted 
men  entering  the  Air  Force  was  developed.  The  inventory,  called  VOICE  for  V^oca" 
tional  and  Occupational  Interest  Choice  Examination,  was  based  upon  3,072  surveys 
returned  by  servicemen  with  at  least  six  months  on-the-job  experience  in  eight 
Air  Force  career  fields  and  an  additional  300  recruits  in  basic  training.  The 
purpose  of  the  effort  was  to  develop  and  validate  a prototype  interest  inventory 
to  be  used  in  the  Guaranteed  Enlistment  Program. 

Four  hundred  occupational  interest  items,  each  representing  an  activity, 
were  written  after  completing  a survey  of  the  Educational  Testing  Service's 
test  collection,  a review  of  relevant  occupational  interest  inventory,  and  an 
examination  of  Air  Force  Manual  39-1  (which  contains  descriptions  of  all  Air 
Force  jobs)  and  the  Dictionary  of  Occupational  Titles.  The  items  were  grouped 
into  four  sections:  (1)  occupations  (90  items);  (2)  work  tasks  (210  items); 

(3)  leisure  activities  (70  items);  and  (4)  desired  learning  experiences  (30 
items).  These  sections  composed  the  interest  inventory. 

Four  job  satisfaction  scales  were  also  developed.  They  were:  (1)  Job, 

which  measured  "intrinsic  satisfaction  with  the  actual  work  activity;"  (2)  Peer, 
which  measured  satisfaction  with  one's  co-workers;  (3)  Supervision,  which 
measured  satisfication  with  one's  supervisor;  and  (4)  Air  Force,  which  measured 
satisfaction  with  Air  Force  working  conditions. 

Starting  with  scales  developed  on  half-samples,  an  elaborate  cross-validation 
technique  described  in  Section  IV  was  used.  Briefly  sumnarized,  a comparison  was 
made  of  the  number  of  "hits"  (i.e.,  correctly  predicting  that  an  individual 
a member  of  a career  field)  achieved  when  using  the  instrument  versus  the  number 
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of  hits  achieved  without  it.  It  was  concluded  that  the  instrument  was  useful 
in  measuring  vocational  interests.  However,  test-retest  reliability  was  unsatis- 
factory, and  the  authors  remark,  "The  actual  correlations  for  each  scale  have  not 
been  presented  since  such  a presentation  might  result  in  an  inaccurate  condemnation 
of  scale  reliabilities.  Estimation  of  scale  reliabilities  should  be  performed  in 
the  future  under  rigid  conditions  so  that  the  estimates  may  be  reported." 

An  appendix  provides  a listing  of  items  in  the  interest  inventory. 


Print 

Conventional  and  individualized  instruction 
Method:  Cross-validation  and  test-retest  correlation. 

Sample  size:  1,537  airmen  for  Sample  1,  1535  airmen 

for  Sample  2. 
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REFERENCE  NUMBER:  2.3.3 

DESCRIBE  ENTRY  BEHAVIOR 

CITATION: 

Mockovak,  W.  P.  Literacy  skills  and  requirements  in  Air  Force  career  ladders 

(Report  No.  AFHRL-TR-74-90) . Lowry  Air  Force  Base,  Colorado:  Air  Force 

Human  Resources  Laboratory,  Technical  Training  Division,  December  1974. 

ABSTRACT: 

Fifty- six  Air  Force  career  ladders  representing  35  career  fields  were  examined 
in  order  to  determine:  (a)  the  reading  requirement  level  (RRL)  of  each  career 

ladder;  (b)  the  reading  ability  level  of  personnel  in  each  career  field;  and 
(c)  whether  a "literacy  gap"  existed  between  the  two  measures.  Subject  matter 
specialists  in  each  career  field  performed  a readability  analysis  of  random 
samples  of  training  materials,  technical  orders,  and  Career  Development  Course 
(CDC)  materials.  Also,  scores  on  Airman  Qualification  Examinations  (AQEs)  were 
converted  to  estimated  reading  grade  levels  by  a validated  conversion  procedure. 

It  was  concluded  that:  (a)  reading  skills  and  requirements  varied  widely 

among  career  ladders;  (b)  in  many  career  fields,  materials  were  not  written  to 
the  reading  ability  level  of  the  user;  and  (c)  as  reading  ability  level  of  per- 
sonnel dropped  in  different  career  ladders,  there  was  no  matching  drop  in  reading 
demands  imposed  on  trainees  and  job  incumbents. 

Reading  requirement  levels  (RRL)  varied  from  a low  of  10.6  in  the  Cook  career 
ladder  to  a high  of  14.0  in  the  Telephone  Switching  Equipment  Repairman  ladder. 

Two  tables  give  RRL  for  the  56  career  ladders  and  for  the  35  career  fields. 

Due  to  constraints,  it  was  not  possible  to  determine  what  effect  job  experience 
had  on  reducing  a job's  reading  demand  or  to  estimate  the  relative  importance  of 
reading  in  career  ladders. 

Reading  grade  level  varied  from  an  average  of  9.1  in  the  Aircraft  Pneudraulic 
Repairman  specialty  to  14.6  in  the  Plrogranmi ng  Specialist  ladder.  The  average 
reading  grade  level  across  the  Air  Force  career  ladders  was  10.8. 

Four  decision  rules  for  determining  "literacy  gaps"  are  given.  They  range 
from  50-50  (50%  of  the  personnel  can  read  and  understand  50%  of  the  reading 
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material  they  encounter)  to  75-85  (85%  of  the  personnel  can  read  and  understand 
75%  of  the  material  encountered).  The  decision  rules  are  applied  to  ten  career 
fields,  and  it  is  was  concluded. that  reading  improvement  programs  should  be  applied 
to  career  fields  with  minor  "literacy  gaps"  while,  for  other  career  fields, 
rewritten  materials  should  be  considered.  The  author  notes  that  the  problem  of 
"literacy  gaps"  is  too  complex  and  has  too  many  implications  for  training  to  be 
discussed  in  detail  in  this  report. 

An  appendix  gives  the  estimate  average  reading  grade  level  of  USAF  personnel 
by  career  ladder. 
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REFERENCE  NUMBER:  2.3.4 

DESCRIBE  ENTRY  BEHAVIOR 

CITATION: 

Vitola,  B.  M. , Mullins,  C.  J., 

and  Brokaw,  L.  D. 

Quality  of  the  all -volunteer 

Air  Force  - 1973  (Report  No 

. AFHRL-TR-74-35). 

Lackland  Air  Force  Base,  Texas: 

Air  Force  Human  Resources  Laboratory,  Personnel  Research  Division,  April  1974. 

JSAS  Catalog  of  Selected  Documents  in  Psychology,  1974,  4,  153.  (Ms.  No.  820) 

ABSTRACT: 

Characteristics  of  aptitude,  race,  age,  education,  and  geographic  area  of  en- 
listment of  all -volunteer  force  enlistees  were  studied.  The  following  conclusions 
were  reached;  (a)  average  aptitudes  of  1973  enlistees  in  the  mechanical  and  elec- 
tronics areas  showed  an  increase  over  1971  and  1972  enlistees;  (b)  average  aptitudes 
of  1973  enlistees  in  the  general  and  administrative  areas  showed  a decrease  when  com- 
pared to  1971  and  1972  enlistees;  (c)  an  appreciable  loss  of  enlistees  occurred 
who  had  completed  post-high  school  education,  which  was  accompanied  by  an  annual 
downward  shift  of  percentages  of  first-term  airmen  scoring  in  the  higher  centile 
levels  of  the  Airman  Qualification  Examination  (AQE);  (d)  compared  to  other  areas, 
the  1973  aptitude  level  patterns  of  South-Southwest  enlistees  were  not  consistently 
low,  as  they  were  in  1970  through  1972;  (e)  1973  aptitude  levels  of  black  enlistees 
were  higher  than  1970  through  1972  aptitude  levels,  across  both  geographic  areas 
of  enlistment  and  total  sample;  (f)  a need  may  exist  to  offer  incentives  to 
high-aptitude  people  in  the  general  and  electronics  areas;  and  (g)  Air  Force 
operational  capability  does  not  seem  to  be  significantly  affected  by  the  zero- 
draft  environment.  The  conclusions  are  based  on  data  from  324,935  male,  nonprior- 
service  basic  trainees.  The  data  source  was  the  Air  Training  Conmand  (ATC)  per- 
sonnel processing  files  on  nonprior-service  accessions. 

Three  additional  points  are  made.  First,  the  removal  of  the  draft  as  a 
motivator  for  enlistment  resulted  in  a decreased  number  of  accessions  completing 
education  beyond  the  high  school  level.  In  1970,  18  percent  of  the  total 
accessions  had  13  or  more  years  of  education;  by  1973,  the  number  dropped  to 
7 percent.  Second,  the  discharge  rate  for  first- term  airmen  after  four  years  ^ 
was  79  percent.  Third,  the  average  difference  on  mean  scores  for  AQE  aptitude 
composites  between  blacks  and  nonblacks  decreased. 

« 
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REFERENCE  NUMBER:  2.4.1 

DETERMINE  SEQUENCIAL  STRUCTURE 

CITATIONS: 

Dansereau,  D.  R.,  Evans,  S.  H.,  Wright,  A.  D. , Long,  G.  and  Atckinson,  T.  Factors 
related  to  developing  instructional  information  sequences:  Phase  I 
(Report  No.  AFHRL-TR-73-51(I)) . Lowry  Air  Force  Base,  Colorado:  Air  Force 
Human  Resources  Laboratory,  Technical  Training  Division,  March  1974. 

Dansereau,  D.  R.,  Eva.is,  S.  H.,  Atckinson,  T.  A.,  and  Long,  G.  L.  Factors  relating 
to  the  development  of  optimal  instructional  sequences  (Report  No.  AFHRL-TR- 
73-51(11)).  Lowry  Air  Force  Base,  Colorado:  Air  Force  Human  Resources 
Laboratory,  Technical  Training  Division,  June  1974. 

ABSTRACT: 

In  Phase  I of  the  study,  an  extensive  review  and  synthesis  of  the  literature 
relating  to  the  impact  of  selected  variables  upon  the  design  and  development  of 
effective  instructional  sequences  were  conducted,  A major  conclusion  drawn  was 
that  the  findings  of  previous  studies  were  consistently  inconsistent.  Some 
studies  found  greater  performance  with  application  of  particular  sequences  while 
others  did  not.  A portion  of  the  inconsistency  was  attributed  to  methodological 
faults.  For  example,  a number  of  studies  reported  that  branching  sequences 
(versus  fixed  sequences)  resulted  in  better  terminal  performance  and/or  less  time 
to  complete  instruction.  Although  branching  was  the  variable  being  manipuUted 
intentionally,  it  generally  led  to  fewer  items  being  presented  in  comparison  to 
the  fixed  sequence.  Thus,  it  was  possible  that  the  evident  improvement  was  due 
in  fact  to  a reduced  number  of  instructional  items  rather  than  branching  £er  se  . 
Further,  a fixed  sequence  with  the  right  items  dropped  may  produce  the  same  results 
as  the  branching  technique. 

Three  other  factors  causing  inconsistent  findings  were:  (a)  lack  of  systematic 

procedures  for  describing  information  structures  and  for  developing  instructional 
sequences;  (b)  lack  of  sensitive  dependent  measures  for  assessing  sequence  effects; 
and  (c)  lack  of  attention  to  individual  differences  in  information  processing 
styles  and  capacity. 
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In  addition  to  the  literature  review  and  analysis,  three  pilot  studies  were 
conducted  in  preparation  for  the  main  experiment  reported  in  Phase  II.  The  pilot 
studies:  (1)  identified  an  appropriate  presentation  rate  for  the  main  experiment 

(45  seconds  per  page) ; (2)  selected  Air  Force  materials  of  differing  difficulty 
for  the  main  experiment;  and  (3)  chose  two  instructional  sequences  plus  the 
original  Air  Force  sequences  for  the  main  experiment. 

The  main  experiment,  reported  in  Phase  II,  was  designed  to:  (a)  assess  the 

utility  of  multidimensional  scaling  (called  INSCAL)  as  a procedure  for  describing 
the  information  structure  of  Air  Force  instructional  material  and  to  investigate 
INSCAL's  potential  for  generating  effective  instructional  sequences;  (b)  determine 
if  information  sequence  requirements  interacted  with  a measure  of  intellectual 
ability,  namely,  the  Delta  Reading  Vocabulary  Test;  (c)  determine  the  effect  of 
two  alternative  presentations  (verbal  and  pictorial/graphic)  on  test  performance; 
(d)  assess  the  interaction  of  instructional  material  difficulty  with  other  indepen- 
dent variables;  and  (e)  determine  the  usefulness  of  two  new  performance  measures, 
a concept- close  procedure  and  a similarity- judgement  methodology. 

Based  on  data  from  180  students,  the  conclusions  were:  (a)  INSCAL  was  useful 

for  defining  information  complexity  of  technical  material,  developing  instructional 
sequences  within  technical  material,  and  providing  an  index  of  expert  inter-rater 
consensus;  (b)  alternative  instructional  sequences  affected  student  performance; 

(c)  small  performance  differences  resulted  when  pictorial  technical  information 
sequences  were  compared  to  verbal  print  sequences;  (d)  increased  technical 
difficulty  of  instructional  materials  decreased  student  performance;  (e)  reading 
aptitudes  were  significantly  related  to  performance  in  both  pictorial  and  verbal 
print  presentation  modes;  and  (f)  instructional  sequences  did  not  interact  with 
student  aptitudes. 
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REFERENCE  NUMBER:  2.4.2 

DETERMINE  SEQUENCE  AND  STRUCTURE 

CITATION: 

Intensification  of  the  learning  process:  Diagnostic  instruments  — Learner  state 

check  list  evaluation  response  form.  Doylestown,  Pa.:  Bucks  County  Public 

schools,  February  1970.  (ERIC  Document  Reproduction  Service  No.  ED  063  345) 

ABSTRACT: 

Intended  for  elementary  public  schools  but  also  useful  for  more  advanced  grades, 
check  lists  for  determining  students'  positions  along  a curriculum  scale  are  pre- 
sented. A behavioral  objectives  evaluation  response  form  is  included.  The  check- 
lists and  response  form  are  designed  to  clarify  such  questions  as:  (a)  should 

behavioral  objectives,  and,  hence,  instructional  materials,  be  grouped  by  grade  or 
across  grade  level  by  subject  matter;  and  (b)  should  behavioral  objectives  be 
grouped  by  curriculum  area?  Clarification  results  from  the  clear  listing  and 
specification  of  learner  states  for  various  curriculum  areas,  and  from  the  response 
form  which  describes  the  extent  to  which  a concept  or  skill  being  taught  is  learned 
and  provides  a means  for  recording  successive  trials  or  responses  until  mastery  is 
achieved  (maximum  number  of  trials  in  this  particular  application  was  4). 

To  determine  how  well  a student  achieves  behavioral  objectives,  the  response 
form  uses  four  numbered  codes.  The  first  code  is  0 (zero)  and  indicates  that  the 
student  is  utterly  unable  to  perform  the  behavioral  objective  performance  require- 
ments. Reevaluation  of  the  diagnostic  input  as  related  to  the  individual's 
prescription  is  needed.  The  second  code  is  1 and  indicates  some  measurable  ability 
to  perform  in  the  manner  prescribed  but  only  after  additional  prompting.  Reevalua- 
tion is  still  necessary.  The  third  code  is  2,  meaning  that  the  learner's  opera- 
tional level  indicates  partial  mastery,  but  requires  strengthening  in  the  specific 
context  of  the  prescribed  objective(s).  Reevaluation  is  still  necessary.  The 
fourth  code  is  3,  indicating  that  the  learner's  operational  level  approaches  the 
optimum  level.  He  is  able  to  achieve  the  performance  level  required  by  the 
objective.  In  fact,  such  outstanding  performance  may  be  achieved  that  reevaluation 
is  needed  in  order  to  restructure  future  objectives. 
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Although  the  above  procedure  is  a simple  approach,  it  may  provide  a good 
manual  backup  for  assessing  student  performance  and  determining  learning  sequences 
in  computer-based  learning  systems. 

DATA  SUMMARY: 
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Print 

Application: 

Conventional  and  individualized  instruction 
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None 
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Key  number: 
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SPECIFY  LEARNING  EVENTS/ACTIVITIES 


No  tools  or  procedures  were  identified  for  this  activity. 
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3.2  Specify  Instructional  Management  Plan  and  Delivery  System 

This  ISD  activity  consists  of  three  principal  steps.  First,  appropriate  media 
for  use  in  the  instructional  system  are  selected.  A systematic  approach  to  media 
selection  requires  consideration  of  the  nature  of  the  objectives  and  of  the  learning 
category  for  each  group  of  objectives,  the  use  of  learning  guidelines  and  the 
learning  activities  required  by  those  guidelines.  Since  instructional  requirements 
vary  for  different  learning  objectives,  equipment  and  faculty  constraints,  cost  of 
procurement  and  replacement,  development  costs,  and  study  characteristics  must  be 
considered.  Second,  a course  management  plan  is  developed.  It  indicates  exactly 
how  instruction  is  to  be  conducted,  how  students  are  to  be  managed  and  tested,  what 
instructors  and  support  personnel  are  to  do,  and  how  all  system  elements  interrelate. 
Third,  a system  management  plan  based  on  these  decisions  is  prepared  which,  in 
addition  to  being  a clear  explication  of  system  policy  and  operation,  assists  in 
internal  evaluation  of  the  system. 

More  than  any  other  ISD  step  in  the  IPISD  model,  specifying  an  instructional 
management  plan  and  delivery  system  hinges  on  decisions  based  upon  policy.  Virtu- 
ally no  decision  is  made  in  this  activity  which  does  not  affect  several  different 
activities,  commit  various  resources,  or  affect  the  utilization  of  personnel  and 
facilities.  The  decision-maker  must  be  aware  of  the  possible  alternatives  to  a 
particular  plan  and  provide— or  be  provided  with--information  about  facilities, 
personnel,  and  other  available  resources.  Decisions  made  in  the  execution  of  this 
ISD  activity  are  based  upon  something  beyond,  or  in  addition  to,  the  evidential 
rightness  or  wrongness  of  a technical  approach.  The  decisions  are  based  upon  con- 
straints, implications  and  priorities.  In  a word,  they  are  based  upon  policy, 
and  policy  cannot  be  procedural i zed  in  the  form  of  a generalized,  transportable 
tool  or  procedure  for  course  developers. 

For  this  reason,  the  references  in  this  section  are  handled  differently  from 
references  in  other  sections.  First  presented  are  six  abstracts  of  documents 
which  are  seen  as  useful  to  the  decision-making  process  in  any  environment  or 
instructional  setting.  Following  that  is  a list  of  citations  only  which  are 
representative  of  the  type  of  literature  which  may  aid  the  decision-making  process 
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that  is  so  much  a part  of  specifying  an  instructional  management  plan  and  delivery 
system.  Whether  or  not  a particular  citation  is  considered  a tool  or  procedure 
depends  upon  the  context  of  the  instructional  or  delivery  system  being  considered. 
These  citations  are  offered  to  provide  some  indication  of  the  types  of  activities 
contained  in  this  ISD  step  and,  for  those  interested  in  a specific  area,  to  provide 
some  leads  to  additional  literature. 
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REFERENCE  NUICER:  3.2.1 

CITATION: 


SPECIFY  INSTRUCTIONAL  MANAGEMENT  PLAN 
AND  DELIVERY  SYSTEM 


Bretz,  R.  The  selection  of  appropriate  communication  media  for  Instruction: 
A guide  for  designers  of  Air  Force  technical  training  programs  (Report 


No.  R-601-PR).  Santa  Monica,  Calif.:  RAND  Corporation,  February  1971. 


ABSTRACT: 

Bretz  defines  communication  media  as  a complete  system  for  the  communication 
of  messages,  which  generally  encompasses  a process  for  the  production  of  programs, 
a process  for  the  delivery  of  these  programs,  and  a means  for  program  display.  The 
primary  input  to  the  system  is  the  message  it  processes.  Secondary  inputs  to  the 
system  consist  of  capital  equipment,  labor,  expendable  materials  for  program 
planning  and  production,  and  engineering  and  administration  functions.  The  primary 
output  of  the  system  is  the  message  that  was  put  in  a suitable  format,  i.e., 
displayed  programs  or  displayed  data.  It  is  but  one  element  of  an  instructional 
system,  since  it  helps  the  instructional  system  to  bring  about  student  learning. 
When  the  primat7  purpose  of  a communication  medium  is  to  cause  learning,  then  it 
may  be  considered  an  instructional  medium.  This  report  addresses  only  "self- 
contained  communication  systems  which  have  reproducable  software;  supplementary 
media  such  as  audio  and  visual  aids,  which  only  assist  a teacher  in  presenting  a 
lesson,  are  not  considered." 


Communication  media,  equipment  configuration,  and  program  content  for  eleven 
different  uses  of  communication  in  instruction  are  discussed:  (a)  providing 

knowledge  of  learning  objectives;  (b)  motivating  the  learner;  (c)  presenting 
information;  (d)  stimulating  discussion;  (e)  directing  learner  activities; 

(f)  conducting  drill  and  practice;  (g)  reinforcing  learning;  (h)  providing  a 
learner-simulator  interface;  (i)  evaluating  learner  progress  and  program  effective- 
ness; (j)  assisting  the  administration  of  instructional  systems;  and  (k)  assisting 
the  research  and  development  function  of  instructional  systems.  A procedure  is 
described  for  selecting  media  based  on  a given  instructional  need.  The  selection 
is  based  on  required  learning  objectives,  subject  matter  content,  learner  popula- 
tion characteristics,  and  the  instructional  method  chosen  (such  as  lecture,  self- 
paced,  or  a combination  of  both).  Four  specific  questions  in  media  selection  are 
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considered:  (a)  is  there  a need  for  visual  presentation;  (b)  is  there  a need  for 
audio  presentation;  (c)  is  there  a need  for  full  motion;  and  (d)  what  media  mix  is 
optimum  for  presenting  abstract  information? 


A comprehensive  glossary  of  terms  used  in  the  report  and  for  media  use  in 
general  is  given. 


DATA  SU^fiARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Audio,  visual,  video 
Individualized  instruction 
None 

Military  (Air  Force) 

3.2  Specify  Instructional  Management  Plan  and  Delivery 
System 
Procedure 
65 
10 
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REFERENCE  NUMBER:  3.2.2 

CITATION: 


SPECIFY  INSTRUCTIONAL  MANAGEMENT  PLAN 
AND  DELIVERY  SYSTEM 


Lonigro,  0.  K.,  & Eschenbrenner,  A.  J.,  Jr.  A model  for  selecting  the  media  in 
technical /vocational  education.  In  H.  Hitchens  (Ed.),  Selecting  media  for 
learning:  Readings  from  Audiovisual  Instruction.  Washington,  D.C.: 

Association  for  Educational  Communications  and  Technology,  1974. 


ABSTRACT: 

A model  is  presented  for  selecting  media  for  technical/vocational  education. 
Five  factors  are  considered:  (a)  types  of  learning  required  for  a job  task; 

(b)  proficiency  levels  required;  (c)  types  of  learning/media  matches;  (d)  profi- 
ciency levels  achieved  by  various  learning  types/media  matches;  and  (e)  relative 
production  costs  of  various  media. 

Two  applications  of  the  model  are  discussed  in  the  area  of  electronics 
technical  training.  Three  steps  are  required  for  applying  the  model:  (a)  identify 

the  type  of  behavior  to  be  learned;  (b)  specify  the  selection  constraints  in  terms 
of  required  proficiency  and  acceptable  cost;  and  (c)  plot  proficiency  which  can  be 
achieved  by  specific  media  versus  costs  for  specific  media. 


A "Gameboard  for  Media  Selection"  is  given.  One  axis  indicates  proficiency 
levels  from  low  to  high.  Another  axis  indicates  production  costs  from  low  to  high. 
Data  for  plotting  the  gameboard  are  given  in  Figures  1 and  2.  Figure  1 has  eight 
rows  for  media  and  five  columns  for  types  of  learning.  The  cells  of  the  matrix 
contain  the  proficiency  levels  which  can  be  achieved  by  using  the  types  of  media. 
Figure  2 gives  the  production  costs  for  eight  types  of  media  in  terms  of  relative 
lows  to  highs.  When  the  required  proficiency  level  for  a type  of  learning  and  the 
acceptable  cost  for  media  production  are  identified,  they  are  plotted  on  the  game- 
board.  A clear  comparison  of  type  of  media  versus  cost  and  proficiency  level 
results. 
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Authoring  aid  type: 
Number  of  pages: 
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Still  pictures/graphs,  motion  pictures,  television, 
simulation,  audio  recordings,  programmed  instruction, 
tape/slide,  computer  assisted  instruction. 

Media  Selection 
None 

Military  (Air  Force) 

3.2  Specify  Instructional  Management  Plan  and  Delivery 

System 

Tool 

4 
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CITATION: 


SPECIFY  INSTRUCTIONAL  MANAGEMENT  PLAN 
AND  DELIVERY  SYSTEM 


National  Security  Industrial  Association.  Survey  of  data-presentation  concepts 
and  techniques.  Washington,  D.C.:  Author,  Technical  Publications  Panel, 

Logistics  Management  Advisory  Committee,  January  1973. 


ABSTRACT: 

One  way  to  reduce  the  growing  maintenance  expense  of  increasingly  complex 
military  systems  and  equipment  is  to  provide  more  effective  technical  documenta- 
tion, i.e. , better  maintenance  procedures  and  reference  manuals,  for  use  by 
technicians  with  little  training  and  experience.  This  report,  in  the  format  of  a 
user's  guide,  describes  new  maintenance  concepts  and  techniques  which,  compared  to 
conventional  techr  a manuals,  usually  results  in  superior  technician  performance. 
"In  general,  this  is  true  for  the  time  required  to  isolate  faults,  the  number  of 
errors  made  during  maintenance,  and  often  the  length  of  the  technician's  training 
period." 


The  new  concepts  and  techniques  are  designed  to  solve  two  kinds  of  problems: 
maintenance/troubleshooting,  and  documentation.  The  second  problem  is  perhaps  of 
more  interest  to  course  developers  who  are  concerned  with  text-handling  and  instruc- 
tional materials  reproduction  methods.  The  report  breaks  the  documentation  problem 
into  eight  areas:  (1)  providing  condensed  detail  level;  (2)  decreasing  manual 

size/bulk;  (3)  improving  access  to  information;  (4)  supplying  more  durable  product; 
(5)  reducing  shipping/storage  requirements;  (6)  facilitating  updating;  (7)  ensuring 
timely  availability;  and  (8)  reducing  costs  of  manual  production. 

The  new  concepts  and  techniques  which  address  the  documentation  problem  are: 

(a)  Computer-Aided  Trouble  Analysis  (CATA)  developed  by  Vought  Aeronautics  Co. 

(b)  Malfunction  Detection,  Analysis  and  Recording  System  (MADAR)  and  its  Ground 
Processing  System  (GPS)  developed  by  Lockheed-Georgia  Co.;  (c)  Condensed  Servicing 
Data  (CONSD)  developed  by  General  Electric  Co.;  (d)  Microfilmed  Maintenance  Manual 
Data  Dissemination  (M3DD)  developed  by  the  3M  Co.;  (e)  Automated  Data  Preparation 
Evaluation  Program  developed  by  McDonnell  Aircraft  Co.;  (f)  SAFEGUARD  Maintenance 
Data  System  (MDS)  developed  by  Western  Electric  Co.;  (g)  Functional  Layout  and 
Presentation  System  (FLAPS)  developed  by  Hughes  Aircraft  Co.;  (h)  Condensed 
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Maintenance  Guides  developed  by  Hughes  Aircraft  Co.;  (i)  Spares  Test  Capability 
Report  developed  by  General  Dynamics  Corp. ; and  (j)  Computer  Graphics  and  Visual 
Module  System  developed  by  William  Fetter  and  currently  used  by  the  Boeing  Company. 

Details  about  each  new  concept  and  technique  were  obtained  from  replies  to  a 
questionnaire  sent  to  member  companies  and  organizations  of  the  National  Security 
Industrial  Association  (NSIA).  Specific  details  about  each  concept  and  technique 
include:  (a)  name  of  concept/technique;  (b)  name  of  developer;  (c)  description  of 

concept- technique;  (d)  operation/maintenance/manual  problems  which  concept  is 
intended  to  solve;  (e)  how  the  concept  solves  the  operation/maintenance/manual 
problem(s) ; (f)  where  the  concept  is  being  used;  (g)  test/study  results,  if  any; 
and  (h)  the  person  to  contact  for  additional  information. 


Since  the  primary  purpose  of  the  report  was  to  collect,  organize,  and 
disseminate  information,  no  concept  or  technique  was  evaluated  by  the  NSIA. 


"The  reader  should  use  this  report  for  a brief  description  of  the  data 
presentation  concept/technique  submitted,  and  contact  the  contributors  and  sources 


in  in  the  report  body. 

if  additional  information  is  desired." 

i SUMMARY: 

Media: 

Print,  Computer-based 

Applicati on: 

Text-handling  and  instructional  materials  reproduction 
systems 

Evaluation: 

None 

Environment: 

Civilian 

Key  number: 

3.2  Specify  Instructional  Management  Plan  and  Delivery 
System 

Authoring  aid  type: 

Procedure 

Number  of  pages: 

153 

Number  of  references: 

None 
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REFERENCE  NUMBER:  3.2.4  SPECIFY  INSTRUCTIONAL  MANAGEMENT  PLAN 

AND  DELIVERY  SYSTEM 

CITATION: 

Pritchard,  R.  D.»  Von  Bergen,  C.,  Jr.,  & DeLeo,  P.  J.  Incentive  motivation 

techniques  evaluation  in  Air  Force  technical  training  (Report  No.  AFHRL-TR- 
74-24).  Lowry  Air  Force  Base,  Colorado:  Air  Force  Human  Resources  Laboratory, 

Technical  Training  Division,  November  1974. 

ABSTRACT: 

Application  of  incentive  motivation  techniques  has  resulted  in  improved 
performance  in  classrooms,  mental  institutions,  and  industry.  This  report 
describes  a study  conducted  at  Chanute  Air  Force  Base  which  evaluated  the  effec- 
tiveness of  incentive  motivation  in  Air  Force  technical  training.  The  incentive 
motivation  program  was  implemented  in  two  courses.  Weather  Observer  and  Aircraft 
Electrical  Repairman.  The  program  was  evaluated  in  terms  of  feasibility,  effect 
on  training  performance,  and  cost-effectiveness. 

Three  incentive  systems  were  implemented  sequentially:  (a)  high  feasibility 

incentives  awarded  on  the  basis  of  performance;  (b)  high  feasibility  incentives 
awarded  on  the  basis  of  effort;  and  (c)  high  and  low  feasibility  incentives  awarded 
on  the  basis  of  effort.  The  fii'st  system's  incentives  consisted  of  commendation 
letters,  time  off,  avoidance  of  work  details,  choice  of  uniform,  and  avoidance  of 
marching  in  formation.  The  incentives  were  dispensed  through  a point  system 
whereby  a student  earned  points  by  improving  his  exam  scores  or  decreasing  his 
time- to- complete.  The  points  could  then  be  spent  on  the  incentives.  The  second 
system's  incentives  were  similar  to  those  of  the  first  system  but  points  were 
awarded  for  effort  rather  than  pure  performance.  A student  was  assigned  a target 
score  according  to  his  ability.  Achievement  above  this  target  score  was  considered 
a reflection  of  effort,  and  points  were  awarded  accordingly.  The  third  system's 
incentives  were  financial  and  consisted  of  U.S.  savings  bonds,  gift  certificates, 
chits  for  various  base  facilities,  and  round  trip  bus  transportation.  Students 
earned  points  by  the  individual  target  score  method. 

Results  of  the  study  indicated  that:  (a)  the  third  system  had  the  greatest 

positive  impact  on  the  time  to  complete  the  Aircraft  Electrical  Repairman  course; 
(b)  none  of  the  incentive  systems  improved  exam  scores;  (c)  all  incentive  systems 
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generally  decreased  remedial  instruction  probations,  and  counselling  sessions  of 
the  Weather  Observer  course;  (d)  all  incentive  systems  produced  increases  in 
reenlistment  intentions  in  both  courses;  (e)  the  incentive  systems  had  neither  a 
positive  nor  negative  effect  on  job  satisfaction  or  overall  attitude  toward  the 
Air  Force;  and  (f)  students  slightly  favored  the  incentive  system  while  instructors 
were  slightly  unfavorable  toward  it,  but  neither  group  found  the  mechanics  of  the 
systems  overly  burdensome. 

Conclusions  of  the  study  were:  (a)  incentive  systems  1 and  2 were  not  effec- 

tive because  the  incentives,  while  highly  feasible  to  implement,  were  relatively 
weak  in  attractiveness;  (b)  implementation  of  system  3 resulted  in  significant 
increases  in  course  performance;  (c)  cost  benefit  analysis  indicated  that  the 
savings  were  more  than  enough  to  offset  the  cost  of  the  incentive  systems  (d)  the 
lack  of  impact  on  the  Weather  Observer  course  was  due  to  the  ceiling  effect  where 
average  performance  was  already  at  a very  high  level;  and  (e)  the  effort-based 
incentive  system  was  not  understood  by  students. 

Implications  of  the  study  for  successful  implementation  of  incentive  motivation 
techniques  in  Air  Force  technical  training  were:  (a)  incentives  must  be  fairly 

powerful;  (b)  increases  in  course  performance  for  trainees  must  not  be  difficult 
to  attain;  (c)  self-paced  courses  are  most  appropriate  for  incentive  techniques; 

(d)  more  frequent  reinforcement  should  occur;  and  (e)  authority  figures  and  peers 
should  provide  positive  social  reinforcement  in  order  to  enable  the  student  to 
achieve  high  performance. 

DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 

Number  of  pages: 

Number  of  references: 


None 

Student  Management 
None 

Military  (Air  Force) 

3.2  Specify  Instructional  Management  Plan  and 

Delivery  System 

Procedure 

294 

173 
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REFERENCE  NUMBER:  3.2.5 

CITATION: 


SPECIFY  INSTRUCTIONAL  MANAGEMENT  PLAN 
AND  DELIVERY  SYSTEM 


Richardson,  F.  C.,  O'Neil,  H.  F.,  Jr.,  Grant,  R.  D.,  & Judd,  W.  A.  Development  and 
preliminary  evaluation  of  an  automated  test  anxiety  reduction  program  for  a 
computer-based  learning  situation.  Austin:  University  of  Texas  at  Austin, 

Computer-Assisted  Instruction  Laboratory,  January  1973. 


ABSTRACT: 

This  report  concerning  an  automated  test  anxiety  reduction  program  Is  divided 
Into  two  parts.  First,  the  literature  Investigating  the  nature  of  test  anxiety 
and  Its  remediation  by  behavior  therapy  techniques  Is  reviewed.  Second,  the 
development  of  an  automated  program  for  the  reduction  of  test  anxiety  In  a computer- 
based  learning  situation  Is  described. 


The  test  anxiety  program  consists  of  four  components.  The  first  component  Is 
a 90-page  manual  written  for  the  program.  The  manual  describes  and  diagrams  the 
process  whereby  attention  Is  directed  to  the  task  (I.e.,  the  test)  Instead  of  to 
oneself,  provides  detailed  Information  about  the  behavioral  and  emotional  dynamics 
of  test  anxiety  and  various  strategies  for  coping  with  It,  and  contains  a series 
of  written  exercises  In  which  the  student  outlines  for  himself  behavioral  strate- 
gies for  coping  with  test  anxiety.  The  second  component  of  the  program  Is  a 
videotape  presentation  of  a female  student  modeling  effective  and  Ineffective 
management  of  test  anxiety.  The  third  component  of  the  program  consists  of  a 
modified  desensitization  procedure  of  deep  muscle  relaxation  and  self-visualization 
of  coping  with  mild  anxiety  In  a graded  series  of  test-taking  scenes.  The  final 
component  of  the  program  requires  that  students  practice  responding  to  test-like 
questions  at  a computer  terminal.  In  addition  to  the  test-like  questions,  the 
terminal  also  presents  Instructions  that  remind  students  of  the  slow,  deep  breath- 
ing technique  for  combating  physical  tension  and  anxiety,  and  of  the  Importance  of 
paying  attention  to  the  test  and  avoiding  self-oriented  worry. 


The  test  anxiety  reduction  program  was  applied  to  16  undergraduate  education 
majors  at  the  University  of  Texas  at  Austin  taking  a required  course  In  Educational 
Psychology  during  the  fall  semester  of  1972.  One  part  of  the  course  was  a sequence 
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of  five  computer-managed  Instructional  modules.  The  results  of  the  program 
Indicated:  (a)  a striking  reduction  In  general  test  anxiety  and  state  anxiety  for 

the  treatment  as  compared  with  control  subjects;  (b)  no  difference  between  the 
module  test  scores  of  the  treatment  group  before  and  after  treatment,  and  no 
difference  between  the  module  test  scores  of  treatment  as  compared  with  control 
subjects  after  the  former  had  completed  the  test  anxiety  program;  (c)  subjects 
expressed  generally  strong  positive  feelings  about  the  program;  and  (d)  subjects 
felt  that  the  manual  written  for  the  program  related  much  useful  Information  con- 
cerning test  anxiety  coping  strategies. 

It  was  noted  that  deep  muscle  relaxation  and  visualization  of  hierarchy  scenes 
occasionally  (though  not  In  this  particular  situation)  give  clients  difficulty 
and  requires  the  assistance  of  a professional.  It  was  recommended  that  a probable 
next  step  would  "explore  eliminating  the  relaxation  and  desensitization  component 
of  the  program  In  favor  of  an  Increased  emphasis  on  practicing  test-taking  tfnd 
anxiety  management  skills  at  the  terminal  on  test-like  materials." 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Envl ronment: 

Key  number: 

Number  of  pages: 
Number  of  references: 


Print  and  videotape 
Student  management 
None 

Civilian 

3.2  Specify  Instructional  Management  Plan  and 
Delivery  System 
41 
74 
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REFERENCE  NUMBER:  3.2.6 

CITATION: 


SPECIFY  INSTRUCTIONAL  MANAGEMENT  PLAN 
AND  DELIVERY  SYSTEM 


S1ege1»  A.  I.,  Federman,  P.  J.,  & Sellman,  W.  S.  A survey  of  student  measurement 
and  course  evaluation  procedures  within  the  Air  Training  Command  (Report  No. 
AFHRL-TR-74-5) . Lowry  Air  Force  Base»  Colorado:  Air  Force  Human  Resources 

Laboratory,  Technical  Training  Division,  July  1974. 


ABSTRACT: 

The  reactions  of  139  training  evaluation,  student  measurement,  and  training 
manager/instructor  personnel  to  the  Air  Force  Instructional  Systems  Development 
(ISD)  system  are  reported.  The  Air  Force  ISD  system  consists  of  five  steps.  The 
first  step  i-s  analysis  of  system  requirements.  Tasks  performed  within  the  overall 
operational  environment  are  identified.  The  second  step  is  definition  of  educa- 
tion or  training  requirements.  The  tasks  requiring  instruction,  the  proficiency 
levels  desired,  and  the  required  resources  are  determined.  The  third  step  is 
development  of  objectives  and  tests.  The  fourth  step  is  planning,  developing,  and 
validating  instruction.  The  fifth  step  is  conducting  and  evaluating  instruction. 
The  general  purpose  of  the  ISD  process  is  to  graduate  students  who  meet  job  perfor- 
mance requirements. 


Although  each  group  generally  regarded  the  ISD  procedures  favorably,  a number 
of  problems  with  the  use  and  implementation  of  ISD  were  identified.  First,  there 
was  a need  for  a more  general  and  extensive  method  for  orienting  supervisory 
personnel  in  the  attributes  and  goals  of  the  ISD  system.  Second,  there  was  a need 
for  more  personnel  who  are  trained  not  only  in  their  individual  assignments  but 
who  also  possess  adequate  insight  into  the  specific  aspects  of  other  ISD  require- 
ments. Third,  there  was  a need  to  reduce  the  time  required  to  apply  the  ISD 
procedures.  Fourth,  there  was  a need  for  streamlining  the  ISD  procedures.  Fifth, 
there  was  a need  for  evaluation  of  hw  well  each  of  the  ISD  steps  is  performed. 

And  sixth,  some  users  elected  to  implement  ISD  through  performance  of  a selected 
subset  of  the  ISD  system  steps.  Since  each  ISD  step  is  dependent  on  a prior  step, 
such  abbreviation  leads  to  faulty  implementation. 
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Student  measurement  techniques  and  course  review  techniques  were  also  surveyed. 
The  general  consensus  toward  each  set  of  techniques  was  favorable  but,  again,  some 
problems  were  identified.  Problems  with  student  measurement  techniques  included: 

(a)  some  written  tests  lacked  reliability:  (b)  standards  for  updating  tests  were 
inconsistent;  (c)  parallel  test  forms  may  have  been  required  but  were  not  available; 
and  (d)  use  of  confidence  and  hierarchical  testing  approaches  were  rare.  The  prin- 
cipal criticisms  of  course  review  techniques  (student,  instructor  and  graduate 
critiques)  were  (a)  their  lack  of  objectivity;  and  (b)  the  interpretation  of  data 
by  personnel  unfamiliar  with  the  total  subject  matter  content  or  with  the  context 
in  which  the  recommendations  were  set. 

Among  the  recommendations  for  program  improvement  are:  (a)  development  of  an 

information  dissemination  program  designed  to  clarify  misconceptions  about  the  ISO 
system  and  to  orient  training  personnel  in  the  attributes  and  goals  of  the  ISD 
program,  (b)  increased  training  for  <personnel  involved  in  ISD  use  and  application, 
(c)  development  of  job  aids  and  work  methods,  along  with  an  increase  in  communica- 
tions efficiency  for  the  purpose  of  "streamlining"  the  ISD  procedures,  (d)  evalua- 
tion of  the  proficiency  with  which  each  ISD  step  is  performed,  (e)  establishment 
of  guidelines  for  updating  tests,  (f)  use  of  alternative  or  parallel  tests  where 
required,  (g)  determination  of  the  predictive  validity  of  tests  used,  (h)  allowing 
sufficient  testing  time  for  the  purposes  on  hand,  and  (i)  development  of  course 
review  techniques  that  can  yield  logitudinal,  objective,  and  quantitative 
information. 

Although  these  results  and  conclusions  may  not  apply  to  other  branches  of  the 
service  or  to  the  civilian  environment,  the  methods  and  procedures  of  this  inter- 
view survey  may  prove  useful  to  other  course  developers  who  are  concerned  with  the 
effectiveness  of  their  particular  materials  development  methodology. 

Appendix  A provides  details  about  the  interview  procedures. 
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ADDITIONAL  CITATIONS  UNDER  REFERENCE  NUMBER  3.2:  SPECIFY  INSTRUCTIONAL  MANAGEMENT 

PLAN  AND  DELIVERY  SYSTEM 

Course  Management-Students 

Beavers,  W.  S.  Effect  of  two  counselling  treatments  upon  the  task-oriented 
behavior  of  seventh  graders.  Paper  presented  at  the  Personnel  and  Guidance 
Association  Annual  Convention,  Atlantic  City,  New  Jersey,  April  1971. 

McKeen,  R.  L.,  & Eisenberg,  T.  A.  On  using  student- generated  sequences  in  the 
development  of  a learning  hierarchy.  Improving  Human  Performance.  A Research 
Quarterly,  1973,  2,  97-106 

Scheuerer,  D.  T.  The  placement  of  students  in  viable  learning  situations  through 
the  use  of  achievement  tests  and  systems  engineering  rather  than  through  annual 
promotion  and  retention.  Volume  II  of  II.  Final  Report.  Boulder:  University 
of  Colorado,  Biological  Sciences  Curriculum  Study,  June  1970.  (ERIC  Document 
Reproduction  Service  No.  ED  057  069) 

Stewart,  J.,  & Clark,  R.  L.  University  of  Maryland  student  scheduling  algorithm. 
Proceedings  of  the  23rd  National  Conference  of  the  Association  for  Computing 
Machinery.  1968,  555-562 

Titmireck,  E.M.  ADVISER  - a program  which  advises  students  on  courses.  Proceedings 
of  the  23rd  National  Conference  of  the  Association  for  Computing  Machinery. 

1968,  535-553 


Course  Management- Instructors 

Devaughn,  J.  E.  A manual  for  developing  reasonable,  objective,  nondiscriminatory 
standards  for  evaluating  teacher  performance.  State  College:  Mississippi 
State  University,  Educational  Services  Center,  September  1971.  (ERIC  Document 
Reproduction  Service  No.  ED  059  151) 

Manis,  D.,  & Solomon,  W.  Using  task  analysis  to  generate  teacher  training 

objectives  and  activities  and  evaluate  their  adequacy.  Paper  presented  at  the 
meeting  of  the  American  Educational  Research  Association,  Chicago,  April  1974. 
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Course  Management- Sup port  Personnel,  Facilities,  and  Equipment 

Defense  Documentation  Center.  Government  acronyms  and  alphabetic  organizational 
designations  used  in  DDC.  Alexandria,  Va.:  Author,  April  1975.  National 
Technical  Information  Service  Reproduction  Service  No.  AD/A  006  800) 

Defense  Research  Analysis  Establishment.  Computer-aided  sequential  testing  for 
equipment  reliability.  Ottawa,  Ontario:  Author,  June  1973.  (NTIS 
Reproduction  Service  No.  AD  919  179) 

Resta,  P.  E.,  & Hanson,  R.  A.  Installation  requirements  for  the  SWRL  first  year 
communication  skills  program.  Evaluation  data,  1968-69.  Inglewood,  Calif.: 
Southwest  Regional  Educational  Laboratory,  June  1971.  (ERIC  Document 
Reproduction  Service  No.  ED  057  993) 

Sullivan,  D.  J.,  & Smith,  E.  A.  Instructional  media  carrel  systems  (Report  No. 
AFHRL-TR-72-48).  Lowry  Air  Force  Base,  Colorado:  Air  Force  Human  Resources 
Laboratory,  Technical  Training  Division,  February  1974. 

Course  Management  - Text-handling  and  Courseware  Reproduction  Systems 

Human  Resources  Research  Organization.  Project  IMPACT:  Computer-administered 

instruction:  Preparing  and  managing  the  content  of  instruction.  IMPACT 

text- handling,  subsystem.  Alexandria,  Va.:  Author,  September  1971.  (ERIC 

Document  Reproduction  Service  No.  ED  055  450) 

National  Security  Industrial  Association.  The  military  technical  manual  industry. 
Washington,  D.C.:  Author,  Logistics  Management  Advisory  Committee,  Technical 
Publications  Panel,  June  1972. 

Course  Management  - Organizational  Interaction 

Andrews,  F.  M.,  & Pelz,  D.  C.  Dimensions  of  organizational  atmosphere  (Analysis 
Memo  No.  7).  Ann  Arbor:  University  of  Michigan,  Institute  for  Social 

Research,  Survey  Research  Center,  February  1961. 
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REFERENCE  NUMBER:  3.3.1  REVIEW/SELECT  EXISTING  MATERIALS 

CITATION: 

Computer  assisted  planning:  A user  manual  for  computer-based  resource  units. 

Buffalo,  New  York:  State  University  College  Buffalo,  Faculty  of  Professional 
Studies,  Research  and  Development  Complex,  October  1972. 

ABSTRACT: 

This  user  manual  describes  computer-based  resource  units  and  gives  directions 
for  obtaining  the  units.  Computer-based  resource  units  are  instructional  packets 
consisting  of  objectives,  content  statements,  activities,  materials,  and  evaluative 
devices  which  are  coded  to  various  learner  variables.  A request  form  and  a listing 
of  approximately  40  unit  titles  are  given. 

Listed  with  applicable  grade  levels,  typical  titles  include:  (a)  Alcohol; 

(b)  Career  Education;  (c)  Communications;  (d)  Drugs;  (e)  First  Aid  and  Survival; 

(f)  Illinois  Test  of  Psycholinguistic  Abilities;  (g)  Measurement;  (h)  Nutrition; 

(i)  Safety  Education;  (j)  Speaking  and  Listening;  and  (k)  Mental  Health. 

For  group  requests,  up  to  five  units  may  be  requested.  For  individual  requests, 
student- teacher  planning  is  encouraged  in  order  to  select  appropriate  units. 

Student  variables  such  as  sex,  reading  level,  mental  age,  chronological  age, 
general  interests,  developmental  tasks,  and  physical  handicaps  may  be  indicated  on 
the  request  forms.  When  the  resource  guides  have  been  received,  an  evaluation  form 
included  with  the  guide  should  be  completed. 

Additional  information  may  be  obtained  from: 

Research  and  Development  Complex 
State  University  College  at  Buffalo 
1300  Elmwood  Avenue 
Buffalo,  New  York  14222 
(716)  862-5433 
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REFERENCE  NUMBER;  3.3.2 

REVIEW/SELECT  EXISTING  MATERIALS 

CITATION; 

Department  of  defense  audiovisual  information  system;  A user  guide  for  the  audio- 
visual products  data  base.  Arlington,  Va.;  Department  of  Defense  Directorate 
for  Audio-Visual  Activities,  Office  of  Information  for  the  Armed  Forces 
(OASD/M&RA),  October  1975. 

ABSTRACT; 

This  detailed  user  guide  is  intended  to  provide  the  required  information  for 
participation  in  the  Defense  Audiovisual  Information  Network,  and  to  utilize  the 
Audio-Visual  Products  Data  Base  in  that  Network.  Instructions,  procedures,  and 
background  information  are  provided  that  cover  the  initial  delivery,  installation, 
and  start-up  operations  of  terminal  equipment  used  to  remote-access  the  centralized 
audio-visual  data  files.  The  user's  guide  is  divided  into  two  sections. 

The  first  section  of  the  report  gives  general  information  including; 

(a)  terminal  installation  of  Hazeltine  equipment;  (b)  terminal  operations  of  the 
Hazeltine  equipment;  (c)  communications  features,  including  Wide  Area  Telecommuni- 
cations Service  (WATS)  service  numbers;  (d)  a list  of  individuals  and  their 
official  responsibilities  for  system  management;  and  (e)  a glossary  of  terms. 

The  second  section  presents  specific  information  about  the;  (a)  concept  of  the 
system,  including  hardware  and  software  configuration;  (b)  file  access  procedures; 
(c)  retrieval  processing;  and  (d)  input  processing,  including  operations  of  the 
on-line  text  editor  and  the  off-line  data  preparation  requirements. 

Several  detailed  appendices  provide;  (1)  input  data  guidelines  for  DD  Form 
1995,  an  official  Department  of  Defense  form  for  data  coding;  (2)  data  descriptions 
(3)  segment  descriptions;  (4)  search  instructions;  (5)  output  options;  (6)  on-  and 
off-line  tabulations;  (7)  tutorial  messages  for  users;  and  (8)  default  conditions. 
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REFERENCE  NUMBER:  3.3.3 

REVIEW/SELECT  EXISTING  MATERIALS 

CITATION: 

Index  of  Army  motion  pictures  and  related  audio-visual  aids  (Pamphlet  No.  108-1). 

Washington,  D.C.:  Department  of  the  Army,  Headquarters,  September  'i3»74. 

Index  of  Army  motion  pictures  for  public  non-profit  use  (Pamphlet  No.  108-4). 

Washington,  D.C.:  Department  of  the  Army,  Headquarters,  May  1975. 

ABSTRACT: 

The  first  report  is  a catalog  with  annotations  of  official  Department  of  the 
Anny  motion  picture  films  and  related  audio-visual  materials,  including  slides, 
audio-visual  kits,  recordings,  and  tape  recordings.  It  is  intended  for  use  by 
instructors,  information  officers,  and  other  film  users  (including  defense  contrac- 
tors) of  the  Army  and  active  Reserves.  Both  classified  and  nonclassified 
materials  are  listed. 

The  second  report  lists  annotations  for  motion  picture  films  for  nonprofit  use 
by  Government  agencies,  civic,  religious,  fraternal  and  educational  groups,  schools, 
colleges,  universities,  and  other  agencies.  Use  of  portions  of  the  films  or  of 
soundtracks  only  is  prohibited,  and  no  admission  fee  may  be  charged  in  connection 
with  showings.  Films  are  shipped  free  of  charge,  via  parcel  post,  and  the  reverse 
side  of  the  mailing  card  is  franked  in  order  that  they  may  be  returned  free  of 
charge.  Films  coded  with  an  "S"  in  the  annotated  entries  may  be  purchased  by 
civilian  organizations. 

The  films  and  related  audio-visual  aids  in  the  first  volume  are  primarily 
oriented  to  Army  career  fields  and  specific  activities.  The  films  in  the  second 
volume  are  concerned  with  items  of  more  general  interest  such  as  Civil  Defense 
preparedness,  information  and  education,  sports,  professional,  medical,  research 
and  development  films.  State-by-state  listings  of  Army  Audio-Visual  Support 
Centers  are  given  in  both  volumes. 
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REFERENCE  NUMBER:  3.3.4  REVIEW/SELECT  EXISTING  MATERIALS 

CITATION: 

National  network  for  curriculum  coordination  In  vocational-technical  education. 

Washington,  D.C.:  Department  of  Health,  Education,  and  Welfare,  Office  of 

Education,  1975. 

ABSTRACT: 

Development,  publication,  and  distribution  of  curriculum  guides  and  various 
curriculum  support  services  are  available  from  a national  network  of  six  regional 
curriculum  management  centers. 

Since  all  regional  centers  operate  under  the  same  charter  from  the  Office  of 
Education  and  provide  essentially  the  same  kinds  of  services,  the  Northwestern 
Vocational  Curriculum  Management  Center,  located  at  Olympia,  Washington,  may  be 
discussed  as  a typical  center.  One  person  appointed  by  the  State  Director  of 
Vocational  Education  Education  from  each  state  served  by  the  center  provides  a 
10-state  Advisory  Committee.  The  Advisory  Committee  sets  operating  policy  and 
emphasizes  the  inventorying  and  cataloging  of  curriculum  and  other  vocational 
materials  currently  in  use  within  the  ten-state  area.  Information  is  also 
collected  about  curriculum  development  activities  currently  underway  so  that 
curriculum  developers  may  become  aware  of  the  work  of  others.  A curriculum  guide 
of  materials  available  for  distribution  within  the  service  area  is  periodically 
published.  Curriculum  entries  in  the  guides  provide  information  about  the  content 
of  the  curriculum  material  packages.  Additional  entries  specify  the  use  level 
(for  example,  "sec,  postsec. ")t  number  and  length  of  pieces  in  the  package,  when 
the  package  was  developed,  whether  the  material  is  listed  in  ERIC,  whether  it  was 
field-tested,  and  ordering  information,  including  cost.  Curriculum  materials  are 
available  in  specific  content  areas  as  well  as  in  more  general  areas  such  as 
individualized  education,  innovative  practices,  guidance,  and  teacher  education. 
Lending  library  listings  are  also  given. 

Curriculum  guides  are  published  in  the  form  of  brochures,  booklets,  and  micro- 
fiche. The  Northwestern  Center  curriculum  guide  is  published  in  newspaper  format. 
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Since  curriculum  material  holdings  of  individual  centers  are  too  extensive  to 
be  usefully  annotated  here,  curriculum  guides  and  additional  information  may  be 


obtained  from: 

William  Reynolds,  Director 
Illinois  Curriculum  Mgt.  Center 
Board  of  Voc.  Ed.  and  Rehabilitation 
1035  Outer  Park  Drive 
Springfield,  IL  62706 

Dr.  Joseph  Kelly,  Director 
Voc.  Curriculum  & Media  Research 
Division  of  Vocational  Education 
225  West  State  Street 
Trenton,  NO  08625 

Dr.  James  Becket,  Director 
Western  Curriculum  Coord  Center 
Vocational  Education  Section 
Dept,  of  Education 
721  Capital  Mall 
Sacramento,  CA  95814 


Dr.  James  E.  Wall,  Asst.  Dean  (R&D) 
Mississippi  State  University 
P.O.  Draw  DX 

Mississippi  State,  MS  39762 

Mr.  Ron  Meek,  Coordinator 
Curriculum  Instructional  Mtrls.  Center 
State  Dept,  of  Voc.  and  Tech.  Ed. 

1515  West  Sixth  Avenue 
Stillwater,  OK  74074 

Mr.  James  L.  Blue,  Director 
Voc.  Curriculum  Management  Center 
Coordinating  Council  for  Occupational 
Education 

Building  17,  Airdustrial  Park 
Olympia,  WA  98504 
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Print,  microfiche 

Conventional  and  individualized  instruction 
Individual  curriculum  packets  may  be  field-tested 
Civilian 
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Tool 

Varies  with  each  center's  publications 
Varies  with  each  center's  publications 
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REFERENCE  NUMBER:  3.3.5 

REVIEW/SELECT  EXISTING  MATERIALS 

CITATION: 

Special  reserve  components  educational  video  tape  catalog  (Pamphlet  No.  350-4). 

Fort  Monroe,  Va.:  Department  of  the  Army,  Headquarters,  United  States  Army 

Training  and  Doctrine  Command,  25  March  1975. 

ABSTRACT: 

This  well-indexed  pamphlet  contains  a listing  of  educational  video  tape 
programs  specifically  selected  for  unit  training.  Primarily  intended  for  use  by 
Reserve  Components,  it  may  be  used  by  other  services.  Army  schools,  and  training 
centers.  Video  tapes  requested  by  users  will  be  copies  furnished  in  3/4-inch 
U-Matic  video  cassettes  or  in  1/2-inch  reels  for  use  with  Sony  Television  Trainers. 
TRADOC  Form  517-R  (contained  in  Appendix  B)  will  be  completed  by  the  user  and 
forwarded  to  the  appropriate  Training  Aids  Support  Office  (TASO).  The  addresses 
and  support  areas  of  specific  TASOs  are  listed  in  Appendix  A.  Loans  are  made  for 
up  to  60  days.  A permanent  retention  program  (loans  in  excess  of  60  days)  allows 
copies  of  tapes  to  be  made  on  user-supplied  raw  cassettes/reels.  Cost  per  30- 
minute  blank  3/4-inch  cassette  is  approximately  $25.10  and  for  a 30-minute  1/2-inch 
blank  reel  is  approximately  $23.50. 

Listings  of  video  tape  programs  are  arranged  by:  (a)  subject  matter; 

(b)  alphabetical  index,  cross-referenced  by  title  or  subject-matter  key  word;  and 

(c)  key-word  in  title,  including  more  than  one  key-word  per  title.  Video  tape 
program  entries  include  the  number,  title,  subject  and  target  audience  (normally 
by  career  specialty). 

The  subject  matter  of  the  video  tape  programs  are:  (1)  Air  Defense;  (2)  Armor; 

(3)  Field  Artillery;  (4)  Aviation;  (5)  Engineer;  (6)  Intelligence;  (7)  Military 
Assistance;  (8)  Military  Police;  (9)  Ordnance;  (10)  Quartermaster;  (11)  Signal; 

(12)  Transportation;  (13)  Women's  Army  Corps;  and  (14)  Non-Proponent. 
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Video  tape 

Conventional  and  Individualized  Instruction 
Individual  programs  may  be  evaluated 
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REFERENCE  NUMBER:  3.3.6  REVIEW/SELECT  EXISTING  MATERIALS 

CITATION: 

Vocational  education.  State  instructional  materials  for  office  occupations. 

Washington,  D.C.:  U.S.  Government  Printing  Office,  1973. 

ABSTRACT: 

An  annotated  listing  of  curriculum  materials  available  from  state  education 
agencies  is  provided  in  order  to  assist  planners,  administrators,  vocational 
educators,  and  course  developers  in  obtaining  existing  curriculum  materials.  This 
particular  title  applies  to  office  education.  Other  guides  in  the  series  are 
available  in  the  following  areas:  (a)  agriculture;  (b)  distributive  education; 

(c)  health  occupations;  (d)  home  economics;  (e)  technical  education;  and  (f)  trade 
and  Industrial  occupations.  They  may  be  obtained  from  the  Superintendent  of 
Documents,  Government  Printing  Office,  Washington,  D.C.  20402.  Requests  for 
specific  material  in  a guide  should  be  addressed  to  the  appropriate  state  education 
agency. 

The  material  in  the  guide  is  arranged  by  coded  entry.  The  first  2-digit 
posi ti on i indicates  the  subject  matter  area  (14,  Office  Occupations).  The  second 
2-digit  position  indicates  the  principal  segment  of  subject  matter  (14.02,  Business 
Data  Processing  Systems  Occupations).  The  third  2-digit  position  indicates  divi- 
sion of  principal  segment  (14.0202,  Peripheral  Equipment  Operators).  The  last 
2-digit  position  indicates  the  first-level  detail  of  division  of  principal  segment 
(14.020201,  Key  Punch  and  Coding  Equipment  Operators).  A discussion  of  the  coding 
system  is  contained  in: 

U.S.  Department  of  Health,  Education,  and  Welfare,  Office  of 

Education.  Vocational  Education  and  Occupations  (OE-80061). 

Washington,  D.C.:  U.S.  Government  Printing  Office,  July  1969. 

Curriculum  materials  listed  in  this  particular  guide  include:  (a)  Accounting 

and  Computer  Occupations  (Bookkeeping);  (b)  Business  Data  Processing  Systems 
Occupations  (Key  Punch  and  Coding  Equipment  Operators,  and  Other);  (c)  Filing, 
Office  Machines,  and  General  Office  (File  Clerks,  General  Office  Clerks,  and 
Other);  (d)  Stenographic,  Secretarial,  and  Related  Occupations  (Stenographers,  and 
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Others);  (e)  Supervisory  and  Administrative  Management  Occupations  (Budget 
Management  Analysis);  (f)  Typing  and  Related  Occupations  (Typists  and  Other);  and 
(g)  Office  Occupations,  Other. 


There  are  two  additional  sections  in  the  guide.  The  first  contains  a listing  of 
more  general  manuals  and  courses  useful  to  administrators  and  teachers,  such  as 
New  Jersey's  Student's  Manual  for  Supervised  Teaching  of  Vocational  Subjects  and 
Oklahoma's  Procedure  for  Establishing  Area  Vocational  Schools.  The  second  contains 
a state- by- state  listing  for  ordering  procedures.  Some  state  education  agencies 
charge  for  publications;  others  send  one  complimentary  copy  of  a curriculum  pack- 
age, and  others  send  everything  free.  Curriculum  developers  in  a particular  state 
should  check  that  state's  order  requirements. 


DATA  SUMMARY: 

Medi  a: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Printed,  programmed  text,  and  audio-visual 
Conventional  and  individualized  instruction 
Individual  states  may  evaluate  their  materials 
Civilian 

3.3  Review/Select  Existing  Materials 

Tool 

25 

None 
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REFERENCE  NUMBER:  3.4.1  DEVELOP  INSTRUCTION 

CITATION: 

Best,  E.  The  development  of  visual  materials  for  CAI  (Technical  Memo  No.  5). 
Austin:  The  University  of  Texas  at  Austin,  Computer-Assisted  Instruction 

Laboratory,  August  1971. 

ABSTRACT: 

Various  procedures  and  materials  used  to  minimize  expense  and  time  in  the 
preparation  and  production  of  computer-controlled  films  are  discussed.  The  films 
were  intended  to  be  used  on  a computer-controlled  rear  screen  projector  which 
could  contain  up  to  1 ,022., different  frames  in  either  color  or  black  and  white. 
Frames  were  addressable  in  the  standard  sound  track  area.  For  text-typed,  one- 
color,  20-frame  filmstrips,  the  cost  of  production  was  approximately  $97  and 
processing  time  was  two  to  three  weeks. 

The  first  step  in  the  production  of  the  filmstrips  was  the  decision  by  the 

author  of  what  portion  of  his  program  to  display  on  the  terminal  output  device  and 

* 

what  portion  to  display  from  the  rear  screen  projector.  The  next  steps  were  to 
meet  with  a media  specialist,  discuss  possible  layouts,  and  approve  a text-frame 
master  for  the  preparation  of  8 x 10-inch  black-and-white  negatives.  Author 
approval  for  the  negatives  was  required,  for  filmstrips,  once  produced,  are  diffi- 
cult to  change. 

Text  masters  were  prepared  on  frosted  acetate.  Artwork  masters  were  done  on 
8 X 10- inch  poster  board,  and  photographed  without  enlargement.  Typing  was  done 
in  an  area  4-1/2  x 6-1/2  inch,  which  enlarged  proportionately  to  8“  x 10"  with  a 
1/2-inch  to  3/4-inch  border.  Typing  compared  favorably  with  pressure-sensitive 
and  photocomposed  letters  in  terms  of  cost  and  speed  of  production.  It  was  noted 
that  no  more  than  nine  lines  of  type  should  be  put  on  a master  if  fast  reading 
is  expected. 

Once  the  8"  x 10"  color-backed  negatives  were  obtained,  they  were  photographed 
in  sequence  on  a light-box.  Inexpensive  and  easily  built,  the  light-box  and  its 
materials  and  dimensions  are  discussed  in  the  text.  The  light-box  was  also  used 
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for  alignment  of  the  negative  on  a clear  piece  of  acetate.  A registration  pen  bar 
was  taped  to  the  top  of  the  box,  and  an  Acme  field  chart  was  used  to  align  the 
negative  on  the  acetate.  This  assured  squared  and  centered  frames,  which  were 
essential  for  accurate  photography. 

For  local  production  shops,  this  report  provides  much  information  required  for 
processing  filmstrips  into  slides  for  either  computer-controlled  rear  screen 
projectors  or  for  transparencies. 

DATA  SUMMARY: 


Media: 

Filmstrips,  slides 

Application: 

Conventional  and  individualized  instruction 

Evaluation: 

None  for  procedure  but  materials  used  were  evaluated 

Environment: 

Civilian 

Key  number: 

3.4  Develop  Instruction 

Authoring  aid  type: 

Tool 

Nunber  of  pages: 

12 

Number  of  references: 

None 
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REFERENCE  NUMBER:  3.4.2 

DEVELOP  INSTRUCTION 

CITATIONS: 

Deterline,  W.  A.,  & Lenn,  P.  D.  Coordinated  instructional  system.  Lesson  book. 
Palo  Alto,  Calif.:  Sound  Education  Inc.,  1972. 

Deterline,  W.  A.,  & Lenn,  P.  D.  Coordinated  instructional  system.  Study  resource 
materials  book.  Palo  Alto,  Calif.:  Sound  Education  Inc.,  1972. 


ABSTRACT: 

This  self-study  course  teaches  a set  of  skills  and  techniques  called  CISTRAIN 
for  designing,  developing,  and  implementing  instruction.  CISTRAIN  means  Coordi- 
nated instruction  System  for  Training.  During  the  full  course  of  instruction, 
the  student  prepares  two  lessons  in  his  subject  matter  area  and  tests  them  on 
other  students.  Course  completion  times  range  from  20  hours  to  50  hours.  Average 
completion  time  is  35  hours  but  one  significant  factor  affecting  that  time  is  the 
size  and  scope  of  the  two  lessons  prepared  by  the  student. 

The  Study  Resource  Materials  Book  contains  all  instructional  articles,  sample 
lesson  materials,  and  procedural  guides  necessary  for  completing  the  course.  The 
Lesson  Book  guides  the  student  through  the  appropriate  portions  of  Study  Resource 
Materials  Book  and  provides  reinforcement  when  required.  Both  books  are  concerned 
with  three  processes  of  instructional  design  and  development:  (a)  lesson  develop- 

ment including  testing  instruments;  (b)  evaluation  and  tryout  of  the  lesson;  and 
(c)  selection  of  the  correct  media  for  lesson  presentation. 

Lessons  are  also  called  study  units.  Evaluation  of  student  performance  on 
each  study  unit  depends  upon  a human  monitor  (called  an  Enabler)  who  conducts 
group  discussions  and  peer  critiques.  There  are  six  components  to  each  study  unit. 
First  is  an  introduction  that  discusses  the  topics,  scope,  and  purpose  of  the 
lesson.  The  next  component  contains  descriptive  objectives  that  discuss  what  the 
student  should  be  able  to  do  at  the  end  of  the  study  unit.  Study  resources  and 
directions  are  then  provided  that  indicate  what  the  student  is  to  do,  what 
resources  are  required,  and  when  and  how  the  instruction  is  to  be  carried  out. 

Next  are  practice  exercises  which  help  students  apply  what  they  have  learned  in 
the  lesson.  Questions  are  asked  and  tasks  are  carried  out.  Discussion  (feedback) 
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of  the  practice  exercise  results  Is  the  next  component  of  a study  unit.  Correct 
responses  for  both  questions  and  tasks  are  shown,  and  explanatory  feedback  is 
given.  The  last  component  of  a study  unit  is  actual  self-evaluation  and< 
discussion,  which  basically  provides  further  practice  and  more  detailed  discussions. 

A glossary  of  29  terms  is  included. 


DATA  SUGARY: 
Media: 

Application: 

Evaluation: 


Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Progranmed  text 

Individualized  instruction 

"The  course  materials  underwent  three  tests  and 

revisions,  and  more  than  one  hundred  faculty  and 

administrators  from  California  Community  Colleges 

participated  in  the  developmental  tests The  final 

revision  was  based  on  data  from  the  final  field  tryout." 
Civilian 

3.4  Develop  Instruction 
Tool 

106  (Lesson  Book),  94  (Study  Resource  Materials  Book) 
None  (both  books) 


1 
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REFERENCE  NUMBER:  3.4.3 

DEVELOP  INSTRUCTION 

CITATION: 

Guidelines  for  developing  computer  based  resources  units.  Revised.  Buffalo: 

State  University  of  New  York,  Educational  Research  and  Development  Complex, 
March  1972.  (ERIC  Document  Reproduction  Service  No.  ED  073  586) 

ABSTRACT: 

Guidelines  for  writing  computer-based  resource  units  are  presented  and 
organized  into  two  operations.  The  first  operation  involves  formulating  behavioral 
objectives,  instructional  content,  activities,  materials,  and  measuring  devices 
for  each  unit  of  instruction,  which  is  termed  a "computer-based  resource  unit." 

The  second  operation  involves  coding  these  five  component  items  for  future  computer 
retrieval.  Every  content  item,  activity,  material,  and  measuring  device  is  coded 
to  a behavioral  objective,  and  each  behavioral  objective  is  coded  to  several 
learner  characteristics. 

Specifications  are  given  for  writing  objectives,  coding  components  to  objec- 
tives, and  coding  objectives  to  learner  variables.  Objectives  written  for 
computer-based  resource  units  must:  (a)  be  behavioral  with  measurable  perfor- 

mance; (b)  be  oriented  to  the  grade  level  of  the  student;  (c)  be  relevant,  based 
on  "some  social  demand,  student  need,  or  philosophical  imperative;"  (d)  call  for 
a single  behavior  in  order  to  minimize  confusion  in  the  evaluation  procedure;  and 
(e)  yet  be  general  enough  so  that  different  approaches  may  be  made  for  accotmo- 
dating  individual  differences  in  students.  Coded  components  have  unique  identifi- 
cation numbers,  beginning  with  "1"  and  continuing  in  consecutive  fashion.  For 
coding  objectives  (and  thus  the  instructional  material  coded  tq  efach  objective)  to 
learner  variables,  an  appendix  of  fourteen  variable  categories  containing  248 
instructional  variables  is  given.  The  fourteen  variable  categories  are: 

(a)  student  interests;  (b)  sex;  (c)  developmental  tasks;  (d)  major  social  func- 
tions, i.e.,  "those  societal  conditions  in  which  all  individuals  must  participate 
at  some  time  in  their  lives;"  (e)  reading  level;  (f)  instructional  activity; 

(g)  suggested  approach  to  instruction;  (h)  taxbnomical  sophistication  of  the 
objective;  (i)  material  descriptor;  (j)  instructional  grouping,  i.e.,  "the  one 
designation  which  would  provide  the  most  effective  environment  for  learning;" 
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(k)  evaluation  devices;  (1)  mental  age  of  the  student  and  chronological  age  of  the 
material  (its  "social  appropriateness"),  which  may  differ  according  to  the  specific 
material;  (m)  physical  handicaps;  and  (n)  learning  environment,  such  as  classroom 
or  field  trip. 


In  addition  to  the  appendix  mentioned  above,  three  other  appendices  provide: 
(a)  a coding  sheet  for  objectives;  (b)  appropriate  categories  for  coding  component 
parts  of  a computer-based  resource  unit;  arid  (c)  a coding  sheet  for  learner 
variables. 


DATA  SUlflARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Computer-based 
Individualized  instruction 
None 
Civilian 

3.4  Develop  Instruction 

Tool 

33 

None 


1 


B-108 


AUTHORING  AID 
BIBLIOGRAPHY 

VOLUME  1 

MDC  El 570 
1 T FEBRUARY  1977 

REFERENCE  NUMBER:  3.4.4 

DEVELOP  INSTRUCTION 

CITATION: 

How  to  design  and  produce  Individualized  instruction  programs.  St.  Paul, 

Minnesota:  Minnesota  Mining  and  Manufacturing  Co.,  Visual  Products  Division, 

1972. 

ABSTRACT: 

This  pragmatic  workbook  is  intended  to  be  used  for  developing  audio-visual 
programs  for  the  3M  Sound-On-Slide  System.  With  minimal  instructional  theory,  the 
following  sample  worksheets  and  forms  are  given: 

1.  Task  Listing.  Task  Listing  Sample,  Action  Verbs  Sample,  Task 
Listing  Worksheet; 

2.  Task  Analysis.  Task  Analysis  Sample,  Task  Analysis  Worksheet; 

3.  Performance  Objectives.  Performance  Objectives  Sample  and  Worksheet; 

4.  Evaluation  Checklist.  Evaluation  Checklist  Sample,  Evaluation 
Checklist  Worksheet; 

5.  Learning  Activities  Guide.  Learning  Activities  Guide  Sample, 

Learning  Activities  Guide  Worksheet; 

6.  Sound-On-Slide  Unit  Design  and  Script.  Story  Board  (Picture 
Descriptions),  Story  Board  Worksheet,  Completed  Story  Board; 

7.  Self-Evaluation.  Instructional  Systems  Development  Evaluation  Form, 
Individualized  Instruction  Materials  Development  Evaluation  Fonn, 
Package  Objectives  Evaluation  Form;  and, 

8.  Development  Schedule.  Instructional  Materials  Development  Schedule 
Worksheet. 

In  keeping  with  the  no-frills  approach,  the  job  area  used  as  an  example  for 
the  development  steps  listed  above  is  Service  Station  Attendant. 


DATA  SUMMARY: 
Medi a: 

Application: 

Evaluation: 

Environment: 


Audio-visual 

Individualized  instruction 

None 

Civilian 
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REFERENCE  NUMBER:  3.4.5 

CITATION: 

Nesbit,  M.,  & O'Neil,  H.  F.,  Jr.  Guidelines  for  editin 

Austin:  University  of  Texas  Press,  1976. 

ABSTRACT: 

Course  authors  need  two  kinds  of  information.  First,  to  communicate  with 
instructional  designers  and  educational  psychologists,  course  authors  need 
accepted  definitions  for  types  of  frames  and  rules  used  in  programmed  instruction 
(PI).  Second,  to  write  effective  programs,  course  authors  need  specific  guide- 
lines for  separating  lesson  content  into  rules  and  examples  which  can  be  integrated 
into  a series  of  logical  frames.  Nesbit  and  O'Neil  present  both  kinds  of 
information. 

The  report's  first  two  sections  define  23  basic  terms  of  PI  (such  as  frame, 
step  size,  and  fading)  and  discuss  three  models  of  PI  (linear,  branching,  and 
autoelucidative).  In  linear  programs,  all  learners  proceed  sequentially  through 
the  same  content  in  the  same  way;  hence,  individualization  occurs  according  to  the 
rate  of  student  progress  through  the  program.  In  branching  programs,  learners 
proceed  along  individual  paths  determined  by  answers  to  multiple-choice  questions. 
Right  answers  move  a student  forward;  wrong  answers  move  him  backward  or  forward  to 
reinforcement  frames  or  remedial  material.  In  autoelucidative  programs,  all  learners 
progress  through  linear  steps  that  also  contain  branching  opportunities.  Large  step 
sizes,  i.e.,  large  amounts  of  text  in  single  frames,  and  multiple-choice  questions, 
allow  students  to  progress  quickly  for  correctly  answering  questions  or  to  reread 
summaries  of  specific  material  for  incorrectly  answering  questions.  The  authors 
present  diagrams  and  examples  for  each  type  of  model. 

The  third  section  of  the  report  presents  guidelines  for  editing  PI  text  which 
allow  authors  to  check  lesson  content  and  style  for  logical  consistency  and  to 
tailor  programs  to  the  student  population.  For  example,  it  is  suggested  that: 

(a)  typed  lines  should  not  be  more  than  ten  words  or  4-1/2  inches  long;  (b)  defi- 
nitions should  be  separated  for  emphasis;  and  (c)  testing  should  be  confined  to 
the  major  issue  of  the  frame. 
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Examples  of  good  and  bad  frames  are  also  given.  Two  typical  errors  In  frame 
design  are  Illustrated:  (a)  a student  Is  not  required  to  respond  to  all  material 

In  a frame,  I.e.,  too  narrow  a response;  and  (b)  a student  Is  given  too  much 
material  to  "digest"  at  once,  I.e.,  too  broad  a stimulus.  The  authors  also 
caution  against  over-  or  under- prompting,  using  page  numbers  as  multiple-choice 
question  alternatives  that  Inadvertently  cue  the  right  answer,  and  using  Indis- 
criminate fading,  which  leads  to  memory,  not  understanding. 

DATA  SUfflARY: 


Media: 

Programmed  text 

AppUcatl  on: 

Individualized  Instruction 

Evaluation: 

None 

Environment: 

Military  (Air  Force) 

Key  number: 

3.4  Develop  Instruction 

Authoring  aid  type: 

Tool 

Number  of  pages: 

35 

Number  of  references: 

14  annotated  references 
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REFERENCE  NUMBER:  3.4.6 

DEVELOP  INSTRUCTION 

CITATIONS: 

Siegel,  A.  I.,  Federman,  P.  J.,  & Burkett,  J.  R.  Increasing  and  evaluating  the 
readability  of  Air  Force  written  materials  (Report  No.  AFHRL-TR-74-28) . 

Lowry  Air  Force  Base,  Colorado:  Air  Force  Human  Resources  Laboratory, 
Technical  Training  Division,  August  1974. 

Siegel,  A.  I.,  Lambert,  J.  V.,  & Burkett,  J.  R.  Techniques  for  making  written 
material  more  readable/ comprehensible  (Report  No.  AFHRL-TR-74-47). 

Lowry  Air  Force  Base,  Colorado:  Air  Force  Human  Resources  Laboratory, 

Technical  Training  Division,  August  1974. 

ABSTRACT: 

In  a previous  survey  of  readability  research,  Siegel,  Federman,  and  Burkett 
(1974)  synthesized  the  findings  most  useful  to  writers.  In  the  second  report, 
Siegel,  Lambert,  Burkett  (1974)  illustrated  the  findings  with  very  good  examples. 
The  topics  covered  include:  (a)  morphemes  (the  number  of  sounds  within  words); 

(b)  center- embedding  (putting  a clause  between  the  subject  and  predicate  of  a 
sentence) 5 (c)  sentence  depth;  (d)  vocabulary  diversity;  (e)  linking  stated 
thoughts  with  intended  conclusions;  (f)  seeing  generalizations;  (g)  cutting  out 
unnecessary  facts  in  diagrams  and  charts;  (h)  repeating  facts;  (i)  organizing 
material;  and  (j)  avoiding  abbreviations.  Several  examples  will  indicate  the 
usefulness  of  the  report. 

Morphemes.  The  more  morphemes  contained  within  a word,  the  harder  the  word  is 
to  understand.  Instead  of  "disassemble  the  linkage,"  write  "take  apart  the 
linkage." 

Center-embedding.  Instead  of  "The  president,  having  no  intention  of  yielding 
to  the  repeated  demands  of  the  citizens,  was  forcing  them  to  revolt,"  write  "The 
president  was  forcing  the  citizens  to  revolt  because  he  had  no  intention  of  yield- 
ing to  their  repeated  demands." 
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Sentence  depth.  Breaking  a long  sentence  Into  shorter  sentences  increases 
readability.  Instead  of  "The  new  car  which  the  neighbors  bought  and  which  has 
already  been  delivered,  is  a gift  for  their  son  who  is  graduating  from  college 
this  semester,"  write  "The  new  car  has  already  been  delivered.  It  was  bought  by 
the  neighbors  as  a gift  for  their  son.  He  is  graduating  from  college  this 
semester. " 

Vocabulary  diversity.  Find  the  words  you  want  to  use  and  stick  with  them. 

A thesaurus  is  heavy  in  more  ways  than  one.  "One  little  piggy  went  to  market, 
one  little  piggy  stayed  home"  is  less  burdening  then  "One  little  piggy  went  to 
market,  one  diminutive  porkster  stayed  home." 

Linking.  Don't  force  your  reader  to  make  his  own  conclusions  or  to  make  a 
guess  as  to  what  goes  with  what.  Instead  of  "Deliver  the  beer  and  the  milk  to  the 
NCO  club  and  the  commissary,"  write  "Deliver  the  beer  to  the  NCO  club  and  the  milk 
to  the  commissary." 

Seeing  generalizations.  In  theory-oriented  instruction,  make  generalizations 
when  necessary.  Instead  of  simply  saying  "The  voltage  in  a circuit  is  equal  to 
the  product  of  the  intensity  of  the  current  and  the  resistance  of  the  current," 
add  "Therefore,  the  voltage  can  be  increased  by  decreasing  resistance  or  by 
increasing  current  intensity." 

Repeating  facts.  Important  facts  should  be  repeated.  Instead  of  saying 
"After  you  have  tightened  the  three  wing-bolts,  attach  the  antenna"  and  then 
dropping  it,  write  "After  you  have  tightened  the  three  wing-bolts,  attach  the 
antenna.  Tightening  the  three  wing-bolts  before  antenna-attachment  simplifies 
later  steps.  If  you  fail  to  tighten  the  three  wing-bolts  before  antenna- 
attachment,  the  result  will  be..." 

Organizing  material.  Provide  memory  aids  for  your  reader  whenever  possible. 
Acronyms  are  memory  aids  (RADAR  for  Radio  Detecting  and  Ranging).  Another  type 
of  memory  aid  is  Psalm  145,  where  the  first  letter  of  each  line  forms  the  alphabet 
sequence.  A third  type  of  memory  aid  is  rhyming  or  jingles  such  as  "Hey  diddle 
diddle,  the  cat  and  the..." 
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Avoiding  abbreviations.  Excessive  use  of  abbreviations  can  paralyze  our 
ability  to  think.  Instead  of  "The  STS,  POI,  and  OSR  are  useful  for  TA,"  write 
"The  Specialty  Training  Standard,  Plan  of  Instruction,  and  Occupational  Survey 
Report  are  useful  for  task  analysis." 


DATA  S^RY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Print 

Conventional  and  individualized  instruction 
None 

Military  (Air  Force) 

3.4  Develop  Instruction 
Tool  (second  report) 

88  (first  report),  28  (second  report) 

35  (first  report),  1 (second  report) 
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REFERENCE  NUMBER:  3.4.7 

DEVELOP  INSTRUCTION 

CITATION: 

Thiagarajan,  S.  The  programming  process:  A practical  guide.  Worthington,  Ohio: 

Charles  A.  Jones  Publishing  Co.,  1971. 

ABSTRACT: 

Thiagarajan  divides  the  prograttming  process  into  five  stages:  task  analysis; 

design;  editing;  developmental  testing;  and  validation  testing.  In  addition  to 
those  stages,  this  self-paced  guide  to  programming  printed  texts  also  considers 
effective  behavioral  objectives  and  various  kinds  of  testing  instruments.  Com- 
pletion time  for  the  program  is  estimated  to  be  40  hours.  The  last  frame  of  each 
chapter  requires  the  student  programmer  to  apply  the  concepts  learned  in  the 
chapter  and  analyze  a topic  of  his  choice.  He  is  required  to  design,  develop,  and 
validate  a program  on  that  topic.  The  resulting  program  is  evaluated  by  using 
specific  checklists  found  in  either  the  main  part  of  the  book  or  in  the  confirmation 
section.  The  total  number  of  points  that  can  be  awarded  for  the  assignments  is  100, 
and  students  scoring  95%  or  more  receive  an  "A"  while  those  scoring  between  90%  and 
94%  receive  a "B".  No  grades  are  indicated  for  other  scoring  ranges. 

A Magic  Square  is  used  as  a consistent  example  for  explaining  task  analysis, 
behavioral  objectives,  and  criterion  tests  (including  entry  tests,  pretests,  and 
posttests).  A Magic  Square  is  a multicellular  square  containing  an  odd  number  of 
cells  with  numbers  placed  in  them  such  that  each  horizontal  row,  each  vertical 
column,  and  each  diagonal  sum  to  the  same  number.  The  numbers  may  be  either 
consecutive  or  increased  by  increments  of  five.  (If  you  want  to  know  how  to  con- 
struct a Magic  Square,  check  p.  18  of  the  text!)  Since  a set  of  rules  and  pro- 
cedures exist  for  creating  the  product,  a task  analysis  of  a Magic  Square  can  be 
done.  Thiar^rajan  explains  task  analysis.  Since  there  is  an  intended  visible  out- 
come, behavioral  objectives  for  a Magic  Square  can  be  constructed.  Thiagarajan 
explains  behavioral  objectives.  Finally,  since  there  are  correct  Magic  Squares 
and  incorrect  Magic  Squares,  criterion  tests  can  be  written  for  Magic  Square  con- 
struction. Thiagarajan  explains  that  also.  It  is  a novel  way  to  explain  task 
analysis,  behavioral  objectives,  and  criterion  tests,  but  the  intricacies  of  the 
Magic  Square  sometime  create  confusion. 
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Editing  frames,  developmental  testing  and  revision,  and  validation  testing 
are  discussed  without  using  the  Magic  Square.  The  discussion  on  editing  is  directed 
toward  external  language  and  subject-matter  editors  to  which  the  student-progranmer 
presumably  has  access.  Emphasis  is  given  to  three  types  of  editing:  (a)  revision 

of  subject  matter  involving  changes  in  language  and  frame  sequence}  (b)  revision 
of  language  not  involving  changes  in  subject-matter  or  frame  sequence;  and  (c) 
revision  of  program  involving  changes  in  language  but  not  in  subject  matter.  In 
the  developmental  testing  and  revisions,  importance  is  placed  on  introductory 
comments  which  help  reduce  students'  test  anxiety,  aversion  to  criticism,  and 
aversion  to  self-instruction.  Validation  testing,  it  is  noted,  collects  and 
evaluates  four  kinds  of  data:  (a)  description  of  the  students  involved  in 

validation  testing;  (b)  indication  of  gain  or  loss  in  the  students'  achievement; 

(c)  time  taken  by  the  students  to  complete  the  program;  and  (d)  the  instructional 
situation  in  which  the  validation  testing  took  place. 

A confirmation  section  at  the  conclusion  of  the  book  contains  correct  responses 
to  the  questions  embedded  in  the  text. 

DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Key  Number: 

Authoring  aid  type: 

Number  of  pages: 

Number  of  references: 


Programmed  text 
Individualized  instruction 
32  students  at  Indiana  University 
3.4  Develop  Instruction 
Tool 
122 
11 


AUTHORING  AID 
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REFERENCE  NUMBER:  3.5.1  VALIDATE  INSTRUCTION 

CITATION: 

Abedor,  A.J.  Development  and  validation  of  a model  for  fonnative  evaluation  of 
self-instructional  multi-media  learning  systems.  Paper  presented  at  the 
meeting  of  the  American  Educational  Research  Association,  Chicago,  April  1972. 

ABSTRACT: 

A revision  model  was  developed  and  field-tested  in  order  to:  (a)  identify 

instructional  deficiencies  through  data  collection;  (b)  analyze  the  deficiencies 
which  require  revision  of  materials;  and  (c)  design.  Integrate,  and  evaluate  the 
revisions.  The  study  was  conducted  in  four  phases.  The  first  phase  was  design 
of  a MK  I flowchart  revision  model  based  on  a review  of  the  literature.  The 
second  phase  was  tentative  evaluation  of  the  I model  based  on  interviews  with 
several  developers  of  multi-media  instructional  treatments.  The  third  phase  was 
to  incorporate  the  results  of  the  interviews  into  a KM  II  revision  model  which 
used  small -group  tryout  and  debriefing  procedures  as  the  primary  technology  for 
identifying  instructional  problems  and  recommending  effective  revisions.  The 
fourth  phase  was  empirical  testing  of  the  MK  II  revision  model  in  three  separate 
experimental  situations. 

Abedor  notes  that  revision  of  materials  used  in  a multi-media  instructional 
system  can  be  a monumental  task,  and  that  persuasive  data  (in  terms  of  both  quantity 
and  quality)  are  needed  to  convince  developers  that  revision  is  required.  "In 
other  words,  the  delivery  vehicle  must  have  several  serious  discrepancies  before 
it  warrants  an  overhaul."  He  also  notes  that  developers  usually  do  not  have  the 
time  or  resources  to  make  multiple,  iterative  revisions  of  materials.  Therefore, 
the  MK  II  model  was  designed  to  generate  large  amounts  of  corroborative  data  on 
instructional  problems  in  a one-shot  trial  of  the  prototype  lesson. 

Unlike  other  formative  evaluation  models,  this  model  uses  face-to-face 
interviews  between  the  lesson  developer  and  small  groups  of  students.  The  optimal 
size  of  the  group  was  determined  to  be  nine  students  plus  the  lesson  developer. 

The  procedures  for  group  debriefings  included  the  following:  (a)  selection  and 

orientation  of  the  volunteer  students  who  vary  in  their  entering  abilities;  (b) 
individual  use  of  the  prototype  lesson  materials  by  these  volunteers;  (c) 
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administration  and  assessment  of  learning  and  attitudinal  measures  to  provide 
data  for  the  debriefing  (d)  development  of  a debriefing  document;  and  (e)  imple- 
mentation of  the  group  debriefing. 

Empirical  testing  of  the  MK  II  model  was  conducted  in  five  field  experiments 
with  three  Michigan  State  University  faculty  members  and  students  in  their  courses. 
The  validity  of  the  MK  II  model  was  defined  as  enabling  the  lesson  developer  to 
identify  unsatisfactory  sequences  of  instruction,  and  as  resulting  in  development 
of  substitute,  satisfactory  sequences.  If  one-third  or  more  of  the  group  of 
students  indicated  dissatisfaction,  boredom,  confusion,  or  lack  of  understanding 
with  a particular  sequence,  then  it  was  deemed  "unsatisfactory."  Experimental  and 
control  groups  were  used  to  validate  the  model. 

In  two  experiments,  significant  differences  were  obtained  (p.  <01 ) favoring  the 
lesson  revised  according  to  the  model  on  all  four  dependent  measures  (posttest, 
gain  score,  percent  achieving  80%  criterion,  and  student  attitudes).  In  the  third 
experiment,  a significant  difference  (p.  <01 ) was  noted  only  on  the  posttest. 

The  author  notes  that,  because  of  social  and  psychological  interactions,  the 
overall  effectiveness  of  the  MK  II  for  data  gathering  will  perhaps  vary  from 
situation  to  situation.  For  example,  "in  the  case  of  developer  B,  the  prototype 
lesson  was  so  ineffective  and  the  derogatory  comment  of  students  so  devastating 
that  by  the  end  of  the  debriefing  he  was  simply  unwilling  to  continue  the  process 
for  the  seemingly  ungrateful  students...  In  the  case  of  developer  C,  he  appeared 
unwilling  or  unable  to  handle  the  relatively  large  number  of  derogatory  comments. 
Thus,  he  closed  off  discussion  prematurely  and  refused  to  revise  the  prototype 
lesson."  In  another  instance,  the  author  notes  that  the  "groups  appeared  to 
develop  a 'momentum'  phenomenon.  If  they  got  started  on  a deragatory  tneme 
they  kept  going  and  the  comments  became  increasingly  derogatory  until  the  author 
was  forced  to  become  defensive  and  begin  justifying  the  lesson  rather  than 
exploring  ways  to  improve  it."  Since  the  effectiveness  of  the  MK  II  model  depends 
on  the  data  gathered  during  group  debriefings,  the  serious  disruptions  undermined 
the  process.  Whether  these  weaknesses  are  a fault  of  the  methodology  or  of  the 
group  dynamics  was  not  determined. 
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The  use  of  Abedor's  model  should  be  explored  since,  unlike  most  models,  it 
attempts  to  function  in  a real-world,  operational  environment.  Further,  data 
seem  to  indicate  that  it  works  effectively  in  identifying  and  generating  useful 
revisions. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


None 

Conventional  and  individualized  instruction 
None 

Civilian 

3.5  Validate  Instruction 

Procedure 

45 

17 
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REFERENCE  NUMBER:  3.5.2 

VALIDATE  INSTRUCTION 

CITATION: 

Aleamoni,  L.M.  MERMAC:  A model  and  system  for  test  and  questionnaire  analysis 

(Research  Rep.  330).  Champaign:  University  of  Illinois,  Office  of 

Instructional  Resources,  Measurement  and  Research  Division,  March  1970. 

(ERIC  Document  Reproduction  Service  No.  ED  055  097) 

ABSTRACT: 

An  evaluation  model  was  developed  and  implemented  which  provides  instructors 
with  basic  data  required  for  decision-making  about  student  abilities,  course 
attitudes,  grade  distributions,  and  test  appropriateness.  The  data  are  provided 
in  useable  formats  and  give  quick  feedback  to  students.  Specifically,  the  model 
is  useful  for  test  analysis  and  course  evaluation  questionnaires  by  allowing  the 
user  six  options. 

First,  test  items  are  scored.  Up  to  forty  subscores  may  be  produced  for  each 
individual,  and  each  item  and  response  may  be  weighted  either  the  same  or 
differently  for  any  subscore.  A sample  output  from  this  SCORE  program  is  given 
in  the  text. 

Second,  scores  for  groups  of  individuals  may  be  analyzed  and  the  following 
descriptions  produced:  (a)  frequency  distribution  and  histogram;  (b)  mean,  median, 

and  standard  deviation;  (c)  Kuder-Richardson  reliability;  (d)  standard  error  of 
measurement;  (e)  Spearman-Brown  prophesy  for  .90  reliability;  (f)  raw  scores, 
standard  scores  may  be  weighted,  and  the  sum  assigned  a letter  grade.  A sample 
output  from  this  TOTAL  program  is  given  in  the  text. 

Third,  a test  item  analysis  printout  may  be  given  to  the  student.  His  test 
score,  a list  of  items  missed  with  his  responses,  and  the  correct  responses  are 
printed.  A sample  of  this  ERROR  program  is  given  in  the  text. 

Fourth,  item  analysis  data  are  provided  as  a plot  of  the  percentage  of 
individuals  responding  to  the  correct  response  by  fifths  of  the  total.  Item 
alternative  analysis  data  are  provided  as  a proportion  of  individuals  responding, 
a point  biserial  correlation,  and  the  number  responding  to  each  alternative  by 
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fifths  of  the  total.  A sample  output  of  this  ITEM  program  is  given  in  the  text. 

Fifth,  item  data  may  be  analyzed  according  to  some  external  criterion  other 
than  the  keyed  test  score.  The  output  format  of  this  SELECT  program  is  the  same 
as  the  ITEM  format. 

Sixth,  data  from  course  evaluation  questionnaires,  attitude  scales,  and  other 
measures  with  no  known  correct  response  may  be  summarized.  A frequency  distribution 
of  responses,  a weighted  mean,  and  a standard  deviation  for  each  item  are  provided. 
Subscores  may  be  generated  with  means,  standard  deviations,  split-half  reliabilities, 
and  percentage  of  individuals  responding  to  the  noncriterion  items.  A sample  out- 
put from  this  QUEST  program  is  given  in  the  text.  The  QUEST  program  has  been 
applied  to  the  Illinois  Course  Evaluation  Questionnaire,  which  yielded  responses  to 
fifty  items  and  provided  six  subscores  concerning  general  course  attitude,  instruc- 
tional method,  course  content,  interest-attention,  instructor  effectiveness,  and 
unique  items  requested  by  a user. 


The  MERMAC  manual  and  system  documentation  can  be  obtained  from  the  University 
Press,  University  of  Illinois,  Urbana,  Illinois  61801. 


DATA  SUMMARY: 

Media: 

Appl ication: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Computer-based 

Conventional  and  individualized  instruction 

None 

Civilian 

3.5  Validate  Instruction 
Tool 
14 
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REFERENCE  NUMBER:  3.5.3 

VALIDATE  INSTRUCTION 

CITATION: 

Bloom,  B.S.  Learning  for  mastery.  Evaluation  Comment.  1968,  1(2) 

Hummel-Rossi , B.  A formative  evaluation  of  Bloom's  mastery  learning  concept. 

New  York:  Van  Valkenburgh,  Nooger,  and  Neville,  1972. 

ABSTRACT: 

This  two-part  evaluation  exercise  is  based  on  the  article  by  Bloom,  and  is 
conducted  in  accordance  with  the  twenty-item,  multiple-choice  test  by  Hummel-Rossi. 
The  test  is  designed  to  assess  understanding  of  the  concepts  given  in  Bloom's  ar- 
ticle and  understanding  the  use  of  formative  evaluation. 

The  article  discusses  the  variables  present  in  a mastery  learning  strategy, 
and  the  approach  used  by  Bloom  to  derive  a mastery  learning  strategy.  There  are 
five  variables  in  a mastery  learning  strategy:  (1)  aptitude  for  particular  kinds 

of  learning;  (2)  quality  of  instruction;  (3)  ability  to  understand  instruction;  (4) 
perserverance ; and  (5)  time  allowed  for  learning.  Bloom  outlines  the  preconditions 
necessary  for  mastery  learning,  the  operating  procedures  required,  and  the  outcomes 
to  be  evaluated.  One  important  precondition  is  a performance  standard  by  which  to 
judge  the  achievement  of  students.  Another  important  precondition  is  formative 
evaluation,  accompanied  by  diagnosis  and  followed  by  prescription.  Outcomes  may 
be  measured  in  both  cognitive  and  affective  domains. 

The  test  is  composed  of  an  instruction  sheet;  a specially- treated  answer 
sheet  which,  when  rubbed,  displays  the  answer;  and  the  testing  instrument.  The 
answer  sheet  (or  card)  provides  immediate  feedback.  If  the  wrong  answer  appears 
in  the  spot  being  rubbed,  then  alternative  choices  are  rubbed  until  the  correct 
answer  appears.  The  whole  testing  package  is  called  a "Trainer-Tester  Self-Scoring 
Student  Response  Device."  A reprint  of  Bloom's  article  is  included  in  the 
package. 

One's  belief  in  the  competency  of  course  developers  and  evaluators  is  usually 
a matter  of  faith.  This  is  one  example  of  an  instrument  which  may  provide  some 
data  concerning  that  competency. 
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DATA  SUMMARY: 

Media:  Print,  chemically-treated  answer  card 

Application:  Conventional  and  individualized  instruction 

Evaluation:  None 

Environment:  Civilian 

Key  number:  3.5  Validate  Instruction 

Authoring  aid  type:  Tool 

Number  of  pages:  11  (article),  4 (test) 

Number  of  references:  18  (article),  none  (test) 
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REFERENCE  NUMBER:  3.5.4  VALIDATE  INSTRUCTION 

CITATION: 

Rayner,  G.  T.  An  empirical  study  of  a methodology  for  the  revision  of  systemati- 
cally designed  educational  material  (Report  No.  24).  Tallahassee:  Florida 

State  University,  Computer-Assisted  Instruction  Center,  March  1972.  (ERIC 
Document  Reproduction  Service  No.  ED  067  877) 

ABSTRACT: 

A revision  model  for  systematically  designed  educational  materials  was  developed 
and  tested.  The  model  can  be  applied  to  a course  which:  (a)  has  been  developed 

using  some  systems  model i and  (b)  includes  several  subcomponents,  including 
specific  objectives  clustered  in  some  "unitized"  way  (conitionly  called  modules), 
instructional  materials,  criterion- referenced  diagnostic  tests,  remedial  prescrip- 
tions, pretests,  and  posttests.  These  components  furnish  the  Information  needed 
for  effective  revisions.  Data  sources  include  both  students  and  subject  matter 
experts.  Measuring  instruments  consist  of  content  examinations,  attitude  scales, 
and  collections  of  informal  and  solicited  student  comments. 

Application  of  the  model  results  in  content  revisions  and  procedural  revisions. 
Content  revisions  can  be  made  on  any  or  all  of  six  components  of  instruction  noted 
above.  A change  in  any  one  component  may  necessitate  changes  in  other  components. 
Procedural  revisions  Include  all  operational  changes  made  in  the  process  of  pro- 
viding instruction  to  students.  Attention  is  focused  on  problems  identified  by 
student  criticisms.  Procedural  revisions  are  mainly  made  in  pretesting,  student 
orientation,  instructional  operation,  posttesting,  and  grading. 

The  revision  model  was  applied  to  a required  course  in  health  education  for 
elementary  education  majors  at  Florida  State  University.  Students  were  enrolled 
in  the  course  for  two  successive  quarters,  and  were  randomly  assigned  to  either 
of  two  treatments:  computer  managed  instruction  (CMI);  or  conventional  instruction 

(Cl).  The  CMI  treatment  was  individualized  while  the  Cl  treatment  was  group-paced 
and  managed  by  a professor.  In  both  treatments,  modularized  instruction  consisted 
of  objectives,  sample  test  items,  selected  instructional  materials,  diagnostic 
tests,  remedial  prescriptions,  and  posttests.  During  the  first  administration  of 
the  course,  only  17  percent  of  the  students  reached  criterion  of  80  percent  or 
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more.  After  the  course  was  revised  according  to  the  model  discussed  here,  71 
percent  of  the  students  reached  criterion.  No  significant  attitude  changes 
occurred  across  the  administrations.  Course  completion  times  and  total  time 
signed-on  to  the  computer  terminal  increased  after  the  revision,  even  though 
the  number  of  course  objectives  dropped  from  32  to  23.  The  number  of  minutes 
logged-on  to  the  terminal  versus  criterion  performance  was  analyzed.  Of  those 
students  spending  200  or  more  minutes  on  the  terminal,  92%  reached  criterion; 
of  those  spending  between  100  and  200  minutes  on  the  terminal,  56%  reached  criter- 
ion; of  those  spending  50  to  100  minutes  on  the  terminal,  73%  reached  criterion. 

After  two  successive  administrations  of  the  health  education  course,  it 
was  recommended  that  the  principal  content  change  in  the  model  itself  should  be 
establishing  more  stringent  criterion  for  test  item  revision.  Since  the  overall  goal 
was  80  percent  mastery,  test  items  with  a posttest  difficulty  level  below  80 
percent  should  be  reviewed  for  ambiguity,  isomorphism  with  the  objective,  clarity 
of  related  instructional  material,  accuracy  of  prescriptions,  and  comments  from 
students  who  missed  the  item.  Three  procedural  changes  were  also  recommended: 

(1)  within-course  reviews;  (2)  small-group  help  sessions;  and  (3)  intra-course 
deadlines  to  prevent  students  from  leaving  the  entire  course  to  the  last  few 
days  or  weeks.  The  need  for  the  last  change  was  indicated  by  the  fact  that,  in 
the  second  administration  of  the  course,  34  percent  of  the  tests  were  taken  in 
the  last  week. 

Debriefing  questionnaires  for  the  CMI  and  Cl  groups  are  given  in  two  appendices. 

DATA  SUMMARY: 

Medi a : 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 

Number  of  pages: 

Number  of  references: 


Computer -assisted  instruction 
Conventional  and  individualized  instruction 
None 

Civilian 

3.5  Validate  Instruction 

Procedure 

163 

29 
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REFERENCE  NUMBER:  3.5.5 

VALIDATE  INSTRUCTION 

CITATION: 

Ward,  M.  E.,  Examination  and  application  of  formative  evaluation  for  author 
utilization  during  the  preparation  of  a CAI  course  (Doctoral  Dissertation, 

University  of  Pittsburg,  1972).  (ERIC  Document  Reproduction  Service  No. 

ED  076  056) 

ABSTRACT: 

Ward  investigated  an  author's  use  of  formative  evaluation  during  the  prepara- 
tion of  a CAI  course.  Four  specific  questions  were  examined:  (1)  what  models 

exist  for  formative  evaluation  related  to  the  development  of  course  materials; 

(2)  of  those  models,  which  one  is  more  appropriate  for  use  during  CAI  course 
preparation;  (3)  using  the  selected  model,  what  results  obtain  for  formative 
evaluation  during  CAI  course  preparation;  and  (4)  what  guidelines  and  forms  can 
be  identified  for  inclusion  in  an  author's  manual  for  formative  evaluation  of  the 
materials  development  process? 

Five  evaluative  models  were  studied.  The  first  model,  developed  by  Stake,  was 
based  on:  (a)  descriptions  of  what  intended  antecedents  or  entry  behaviors  were 

expected,  what  intended  instructional  processes  were  planned,  and  what  outcomes 
would  be  evaluated;  (b)  descriptions  of  what  actually  occurred  versus  what  was  in- 
tended to  occur;  and  (c)  two  evaluative  judgements,  one  subjective  by  an  evaluator's 
personal  standards  and  the  other  objective  by  comparison  of  the  particular  program 
to  alternative,  similar  programs.  Briggs  proposed  a second  model,  the  procedures 
of  which  would  take  effect  during  the  development  and  evaluation  phases  of  the 
material  development  cycle.  Instructional  materials  would  be  selected  and  modified 
on  the  basis  of:  (a)  learner  characteristics;  (b)  intended  outcomes;  (c)  learning 

alternatives  such  as  dual -track  programs;  and  (d)  performance  test  results  from 
small-group  tryouts  and  classroom  tryouts.  In  a third  model.  Baker  and  Schultz 
outlined  seven  steps,  or  components,  in  the  instructional  development  process. 

They  are:  (1)  formation;  (2)  instructional  specification;  (3)  item  tryout;  (4) 

product  development;  (5)  product  tryout;  (6)  product  revision;  and  (7)  operations 
analysis,  in  order  that  effective  procedures  may  be  clearly  identified.  The  fourth 
model,  by  Abedor,  (Reference  5.5.1)  relied  on  technical  review  by  subject  matter  ex- 
perts, group  tryouts,  and  face-to-face  debriefings  between  students  who  took  the  lessons 

B-127 


MCOOMMmt.1.  DouoLJta  j^mrmoMMuncm  coAfMuwt'-CAsr 


AUTHORING  AID 
BIBLIOGRAPHY 


VOLUME  I 


MDC  El 570 
1 1 FEBRUARY  1977 


and  authors  who  developed  them.  Glass  proposed  an  evaluation  model  for  finished 
products.  Tne  model,  prepared  for  use  by  outside  evaluators  in  evaluating  finished 
instructional  products  already  on  the  market,  considered  nine  points:  (1)  the  pro- 

duct description;  (2)  goals  evaluation;  (3)  point  of  entry  of  the  evaluator J (4) 
trade-offs;  (5)  comparative  cost  analysis!  (6)  evaluation  of  technical  quality, 
content,  medium  and  availability;  (7)  evaluation  of  outcomes;  (8)  summative  judge- 
ments and  recommendations;  and  (9)  scope  and  value  claims  modifying  the  summative 
judgements. 

After  consultation  with  the  dissertation  committee.  Ward  decided  to  follow  the 
Baker  and  Schultz  model  and  add  to  it  an  outside  evaluation  component  at  the 
Instructional  Specification  stage.  The  model  was  applied  during  the  development 
by  the  author  of  a CAI  course  for  regular  classroom  teachers  in  education  of 
visually-handicapped  children.  The  course  was  delivered  to  rural  school  teachers 
via  a mobile  CAI  laboratory.  Early  evaluation  comments  indicated  the  need  for  a 
handbook  to  accompany  the  on-line  lesson  materials,  and  so  a handbook  was  written. 
Fifteen  college  students  comprised  the  field  test  for  the  course.  Revisions  to  the 
on-line  course  and  the  handbook  were  based  on:  (1)  student  response  records; 

(2)  on-line  time  records  for  each  chapter  and  for  the  total  course;  (3)  print-outs 
of  unanticipated  responses  to  course  questions;  (4)  print-outs  of  student  comments 
made  on-line  by  students  who  had  branched  themselves  into  the  comment  routine;  and 
(5)  results  of  the  final  examination.  The  Operations  Analysis  phase  of  the  model 
identified  points  where  critical  time  delays  occurred,  where  problems  developed, 
where  specific  activities  facilitated  the  development  process,  and  what  forms  and 
procedures  would  be  useful  for  transfer  to  other  developmental  projects. 

Four  procedures  and  forms  were  identified  which,  according  to  the  author, 

"may  be  useful  to  other  CAI  course  authors  preparing  course  material  for  the  [IBM] 
1500  system  while  at  the  same  time  conducting  formative  evaluation.  Some  discussion 
of  these  procedures  and  forms  might  be  appropriate  to  include  in  a manual  for 
authors  of  CAI  materials."  The  procedures  and  forms  are  as  follows. 

First,  the  Baker  and  Schultz  cycle  for  instructional  materials  development 
provided  an  appropriate  framework  for  scheduling  activities  for  course  development. 
The  model  served  the  purposes  of  formative  evaluation  very  satisfactorily. 
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Second,  time  spent  on-line  learning  the  capabilities  of  the  1500  system  gave 
the  author  ideas  about  how  system  capabilities  could  be  used  effectively,  about  the 
disadvantages  and  advantages  of  certain  formats  for  content  presentation  and 
queries,  and  about  various  forms  of  and  reaction  to  feedback  for  both  correct  and 
incorrect  responses.  This  is  a simple  idea  but  how  many  times  is  it  overlooked? 
When  an  auth<?r  and  a software  programmer  work  in  consort  to  develop  a CAI  lesson, 
the  author  should  spend  the  maximum  permissible  time  on-line. 

Third,  author  revisions  done  prior  to  submitting  lesson  materials  to  the 
CAI  Laboratory  eliminated  changes  and  corrections  which  would  otherwise  have 
consumed  time  either  on-line  or  in  discussion  with  a software  programmer  in  order 
to  clarify  the  author's  intent. 

Fourth,  outside  evaluation  was  facilitated  by  use  of  conment  cards  devised 
by  the  author.  The  cards,  with  spaces  for  identifying  the  lesson  label  and  medium, 
reduced  the  author's  time  needed  to  identify  the  material  being  evaluated  and 
commented  upon.  The  comment  cards  were  chosen  by  Laboratory  personnel  to  replace 
blank  cards  previously  used,  which  was  another  indicator  of  usefulness.  Comment 
card  Wmats  are  given  in  Appendices  G-2  and  G-3  of  the  report. 

Examples  of  author  format  sheets,  audio  message  program  sheets,  and  a CRT 
coding  s'leet  for  development  of  CAI  lessons  are  given  in  Appendix  I of  the  report. 

DATA  SUMMARY: 

1 

Med' a: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 

Number  of  pages: 

Number  of  references: 


Computer-assisted  instruction 
Individualized  Instruction 

Formative  evaluation  by  twelve  outside  evaluators 
Civilian 

3.5  Validate  instruction 

Procedure 

146 

41 
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REFERENCE  NUMBER:  4.1.1  IMPLEMENT  INSTRUCTIONAL  MANAGEMENT  PLAN 

CITATION: 

Air  Force  Advanced  Instructional  System.  Weapons  Mechanic  implementation  plan 

(Revision  4).  St.  Louis,  Mo.:  McDonnell  Douglas  Astronautics  Company  - 

East,  30  September  1975. 

ABSTRACT: 

An  implementation  plan  is  a detailed  document  covering  every  aspect  of 
conducting  and  administering  a course  of  instruction.  This  particular  plan  was 
written  for  a course  in  a computer-assisted,  computer-managed,  multi -media,  multi - 
track  prototype  instructional  system  being  developed  for  the  Air  Force  Air  Training 
Command.  Topics  covered  by  the  Weapons  Mechanic  implementation  plan  include: 

(a)  development  and  validation  of  instructional  materials;  (b)  manual  management 
procedures;  (c)  development  and  validation  of  entry  testing  procedures;  (d)  sequence 
and  structure  of  lesson  materials,  including  alternative  learning  strategies;  (e) 
resource  requirements  for  instruction;  (f)  media  overlap,  multi-track,  and  computer- 
assisted  instruction  strategies;  (g)  incentive  management  procedures;  (h)  acceptance 
tests  for  instructional  materials;  (i)  computer-managed  instruction  capabilities 
and  application;  (j)  media,  media  devices,  and  individualized  carrels;  (k)  main- 
tenance, testing,  and  procurement  of  media  devices;  and  (1)  facility  modification 
and  hardware/equipment  support. 

Since  the  specific  detail  in  an  implementation  plan  cannot  be  widely  trans- 
ferred as  an  authoring  tool  or  procedure,  this  discussion  will  center  on  three 
characteristics  of  value  to  other  course  developers:  (1)  the  kind  of  detail 

required  in  an  implementation  plan;  (2)  the  sequential  treatment  of  instructional 
components;  and  (3)  the  support  documentation  accompanying  the  narrative  text. 

First,  the  detail  in  an  implementation  plan  consists  of  concrete,  specific 
statements  which  allow  personnel  to  implement  the  intended  course  and,  more  impor- 
tantly, to  identify  possible  flaws  in  that  implementation.  For  example,  the 
Weapons  Mechanic  implementation  plan  specifies  the  exact  levels  of  review  for 
instructional  materials  and  in  what  functional  departments  lesson  reviewers  work. 

In  defining  resource  requirements  for  large-group  tryouts,  the  implementation  plan 
specifies,  "The  primary  resource  which  shall  be  required  for  large-group  tryouts  is 
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instructors.  The  procedures  for  using  instructors  to  collect  data  for  dependent 
variables  and  criteria  are  described  in  the  Section  on  Manual  Management  Pro- 
cedures. At  least  2 instructors  per  block,  shift,  and  week  will  be  required. 

Their  roles  are  described  in  the  Section  on  Manual  Management  Procedures."  And 
concerning  the  maintenance  of  carrels,  the  implementation  plan  states,  "Based  on 
the  nature  of  carrels  (e.g.,  no  moving  parts,  no  primary  optical  alignments,  etc.) 
preventative  maintenance  shall  be  performed  on  a 40-day  cycle. . .Preventative 
maintenance  shall  include  inspection  for  surface  damage,  structural  damage  and 
rigidity.  Figure  3-1  shows  the  logic  tree  for  preventative  maintenance."  In 
other  words,  the  clearly  stated  procedures  and  requirements  in  an  implementation 
plan  are  intended  to  prevent  the  confusion  and  unintentional  mistakes  that  result 
from  unplanned  initiation  of  an  instructional  system  or  its  component  parts. 

Second,  by  using  a systematic,  sequential  treatment  of  component  parts,  very 
few  details  are  "forgotten"  or  buried.  For  example,  in  this  document,  the  section 
on  carrels  reads,  "3.2  Carrels,  3.2.1  Implementation  of  Carrels,  3.2.2  Maintenance 
of  Carrels,  3.2.3  Quality  Control  of  Carrels,  3.2.4  Testing  of  Carrels."  Other 
sections  follow  the  same  format.  It  may  read  mechanically,  but  the  thoroughness 
and  detail  assured  by  such  a format  are  required  in  an  implementation  plan. 

Third,  the  support  documentation  accompanying  an  implementation  plan  is 
designed  to  answer  possible  questions  raised  in  the  text  and  to  provide  a factual 
basis  for  the  scheme  of  the  plan.  For  example,  in  this  document  it  is  stated  that 
"During  the  initial  tryout  period,  three  methods  are  explained  in  the  text.  One 
method  is  by  personal  interview,  for  which  Appendix  A contains  a structured 
questionnaire."  Similar  documentation  in  the  form  of  figures  and  tables  also 
exists  for  course  outline,  small-group  tryout  schedule,  facility  modification, 
computer  terminal  installation,  and  other  activities  discussed  in  the  text. 

An  effective  implementation  plan  will  have  these  three  characteristics  of 
great  detail,  logical  treatment  of  component  parts,  and  support  documentation. 

Once  a complex  operation  like  implementing  an  instructional  system  is  "nailed 
down"  with  an  implementation  plan,  system  developers  and  system  users  have  a 
common  ground  for  reviewing  and  refining  the  system. 
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DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Computer-assisted  instruction,  audio-visual,  and 
programmed  text 
Individualized  instruction 
None 

Military  (Air  Force) 

4.1  Implement  Instructional  Management  Plan 

Procedure 

75 

None 
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REFERENCE  NUMBER:  4.2.1 

CONDUCT  INSTRUCTION 

CITATION: 

Air  Force  Advanced  Instructional  System.  Learning  center  instructor  (LCD. 

St.  Louis,  Mo,:  McDonnell  Douglas  Astronautics  Company  - East,  1 March  1976. 

ABSTRACT: 

A self-paced,  individualized,  instructor- training  package  was  written  for  use 
in  a computer-assisted,  computer-managed,  multi-media,  multi-track  prototype 
instructional  system  being  developed  for  the  Air  Training  Command.  The 
package  is  divided  into  two  parts.  The  first  part  consists  of  textual  discussion 
concerning  the  instructor  functions  in  the  Advanced  Instructional  System  (AIS). 

The  second  part  consists  of  performance-orinented  internship  and  certification 
training  documents  for  three  specific  courses:  (a)  Inventory  Management  and 

Materiel  Facilities;  (b)  Precision  Measurement  Equipment;  and  (c)  Weapons  Mechanic. 

The  first  part  of  the  package  contains:  (a)  a glossary  of  terms  related  to 

computer-based  instruction;  (b)  an  introduction  to  the  AIS;  (c)  a discussion  of  the 
functions  and  responsibilities  of  learning  center  instructors;  (d)  details,  pro- 
cedures, and  input  forms  for  the  computer-managed  portion  of  the  system;  (e)  a 
discussion  of  how  to  maintain fevorable  learning  conditions  in  the  learning  center; 
and  (f)  a discussion  of  programmed  texts  and  audio-visual  modules  for  instructional 
use  in  the  AIS. 

The  second  part  of  the  package  is  a procedural i zed  guide  to  instructor  duties 
in  a learning  center.  Topics  covered  include:  (a)  computer  terminal  operations 

for  grading  and  for  learning  center  management;  (b)  instructional  preparations  and 
student-centered  activities  in  learning  center  operations;  (c)  entry  testing  pro- 
cedures; (d)  administrative  duties  required  of  learning  center  instructors;  and 
(e)  curriculum  development  responsibilities  for  lesson  materials  reviewing  and 
career  progression. 
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- 

Media: 

Print 

Application: 

Individualized  instruction 

Evaluation: 

None 

Environment: 

Military  (Air  Force) 

Key  number: 

4.2  Conduct  Instruction 

Authoring  aid  type: 

Procedure 

Number  of  pages: 

70+ 

Number  of  references: 

None 

B-134 

MCooMMm LM.  otHjatjtm  Jkmrmomjkuncm  coavmuvv-  m^mr 


MDC  El 570 
1 FEBRUARY  1977 


AUTHORING  AID 
BIBLIOGRAPHY 

VOLUME  1 

MDC  El 570 
11  FEBRUARY  1977 

REFERENCE  NUMBER:  4.2.2 

CONDUCT  INSTRUCTION 

CITATION: 

Askren,  W.  B.,  & Valentine,  R.  I.  Value  of  job  experience  to  teaching  effective- 
ness of  technical  training  instructors  (Report  No.  AFHRL-TR-70-8) , 

Wright  Patterson  Air  Force  Base,  Ohio:  Air  Force  Human  Resources  Laboratory, 

Training  Research  Division,  June  1970.  (NTIS  Document  Reproduction  Service 
No.  AD  709  876) 


ABSTRACT: 

Is  teaching  effectiveness  related  to  field  job  experience?  After  comparing 
Air  Force  technical  training  instructors  with  and  without  field  experience,  the 
authors  found  no  evidence  of  significant  differences  in  teaching  effectiveness 
between  the  two  groups  of  instructors.  However,  grades  for  students  who  had 
field-experienced  instructors  increased  from  start  to  finish  of  course,  while  grades 
for  students  who  had  non-field-experienced  instructors  decreased  in  the  same  period. 

Both  groups  of  instructors  were  also  rated  in  twelve  categories  by  supervisors. 
Field-experienced  instructors  were  generally  rated  higher  than  non-field-experienced 
instructors  in  three  categories:  (a)  Knowledge  of  Subject  Matter;  (b)  Student 

Interest;  and  (c)  Student  Participation.  No  difference  was  noted  in  the  instructor 
ratings  for  four  categories:  (a)  Lesson  Plan  Preparation;  (b)  Organization  of 

Material;  (c)  Use  of  Training  Aids;  and  (d)  Evaluation  of  Student  Progress. 


Interaction  between  type  of  instructor  and  course  unit  indicated  that  non-field- 
experienced  instructors  were  more  effective  with  lecture-theory  units.  Field- 
experienced  instructors  were  more  effective  with  equipment-oriented  units. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


None 

Conventional  instruction 
None 

Military  (Air  Force) 

4.2  Conduct  Instruction 
Procedure 
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REFERENCE  NUMBER:  4.2.3  CONDUCT  INSTRUCTION 

CITATION: 

Popham,  W.  J.,  & Baker,  E.  L.  Classroom  instructional  tactics.  Englewood 
Cliffs,  N.J.:  Prentice  Hall,  1973 

ABSTRACT:  \ 

Learning  and  attitude  are  closely  related.  Since  instructors  wish  to  promote 
reasonable  attitudes  in  students  (and  thus  create  a better  environment  for  learning), 
repressive  techniques  in  class  management  should  be  avoided.  Popham  and  Baker 
suggest  six  rules  for  rational  classroom  management  based  on  educational  and 
psychological  research.  ^ 

Rule  No.  1.  Attend  to  observable  behavfor.  Criteria  for  judging  the 
propriety  of  students'  actions  should  be  established,  of  course,  but  simply 
because  you  "sense  trouble"  is  seldom  a reason  for  punishment. 

Rule  No.  2.  Tell  students  what  type  of  behavior  is  expected  of  them.  This 
information  should  not  be  expressed  in  a threatening  manner  nor  should  it  be 
hidden  and  then  released  bit  by  bit  when  a student  is  caught  misbehaving. 

Rule  No.  3.  Do  not  reward  undesirable  behavior. 

Rule  No.  4.  Avoid  punishing  undesirable  behavior. 


Rule  No.  5.  Provide  acceptable  alternative  student  responses  for  undesirable 
behavior.  Rules  3,  4,  and  5 are  closely  linked  to  each  other.  The  idea  behind 
them  is:  if  you  want  a behavior  to  continue,  reward  it;  if  you  want  a behavior  to 
disappear,  ignore  it  and  offer  an  alternative  response. 


Rule  No.  6.  Reward  appropriate  behaviors.  When  a student  has  substituted  or 
has  begun  to  substitute  a desirable  response  for  an  undesirable  response,  reward  him. 
An  appropriate  reward  — called  a positive  reinforcer  --  dep-;nds  upon  the  context 
of  the  individual  student  and  the  learning  situation. 
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The  rules  are  presented  in  a printed,  self-instructional  program  complete  with 
an  answer  mask  and  a mastery  test  with  answers.  A coordinated  filmstrip-tape 
program  is  also  available  from  Vincet  Associates,  Inc.,  P.O.  Box  24714,  Los  Angeles, 
California  90024. 

DATA  SUMMARY: 


Media: 

Programmed  text  (audio-visual  is  optional) 

Application: 

Conventional  and  individualized  instruction 

Evaluation: 

None 

Environment: 

Civilian 

Key  number: 

4.2  Conduct  Instruction 

Authoring  aid  type: 

Tool 

Number  of  pages: 

158 

Number  of  references: 

None 
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REFERENCE  NUMBER:  5.1.1  CONDUCT  INTERNAL  EVALUATION 

CITATION:  Bailey,  R.  W.  Testing  manual  procedures  in  computer-based  business 

information  systems.  In  W.  B.  Knowles,  K.  S.  Sanders,  & F.  A.  Muckier  (Eds.), 
Proceedings  of  the  16th  Annual  Meeting  of  the  Human  Factors  Society,  1972, 

395-401 

ABSTRACT: 

Testing  of  manual  procedures  as  an  essential  part  of  the  total  systems 
development  process  is  discussed.  The  Bell  System  personnel  subsystem  testing 
program  is  used  as  an  example.  It  consists  of  six  steps.  The  first  step  is 
Preliminary  Evaluation,  which  consists  of  design  verification  of  personnel  sub- 
system products  by  checklists  and  low-level  simulation  techniques. 

The  second  step  is  a Position  Package  Test  which  evaluates  position  packages 
by  using  representative  subjects  in  a simulated  operational  environment.  A 
position  package  consists  of  position  practice,  training  materials,  performance 
aids,  forms  and/or  information  displays,  and  a statement  of  the  minimum  qualifica- 
tions required  of  position  holders.  Position  may  be  oefined  as  job  responsibility. 

The  third  step  is  a Subsystem  Test  which  analyzes  the  proper  information  flow 
for  processing  every  transaction  that  enters  a system  and  triggers  system  processing. 
Sequential  information  flow  into  and  out  of  positions  is  also  considered. 

The  fourth  step  is  a System  Test,  consisting  of  performance  tests  for  the 
personnel  subsystem  and  the  computer  subsystem  in  separate  simulated  operational 
environments. 

The  fifth  step  is  a Total  System  Test.  The  total  business  information  system 
is  performance-tested  in  a simulated  operational  environment. 
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The  sixth  step  is  a Trial.  The  final  step  of  the  testing  program  is  conducted 
in  a real-time  operational  environment  to  determine  if  the  system  achieves  specified 
system  performance  objectives  under  normal  operating  conditions. 


A test  report  containing  quantitative  results  and  reconriendations  records  not 
only  the  number  and  kinds  of  errors  made  during  the  test  but  also  the  probable 
causes  for  the  errors.  Accuracy  objectives  are  set  before  testing  begins,  and 
estimated  error  rates  derived  from  the  tests  are  compared  with  the  accuracy 
objectives  to  determine  what  changes  are  required  to  make  the  system  meet  its 
performance  objectives. 


The  author  notes  that  errors  can  be  caused  by  seven  factors:  (a)  design;  (b) 

documentation;  (c)  training;  (d)  personal;  (e)  source  data;  (f)  environmental; 
and  (g)  man-machine  interface. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 
Authoring  aid  type: 
Number  of  pages: 
Number  of  factors: 


Simulation 

Conventional  and  individualized  instruction 
None 

Civilian 

5.1  Conduct  Internal  Evaluation 

Procedure 

7 

None 
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REFERENCE  NUMBER:  5.1.2 

CONDUCT  INTERNAL  EVALUATION 

CITATION; 

Miller,  G.  G.,  and  Sellman  W.  S.  Development  of  psychometric  measures  of  student 
attitudes  toward  technical  training:  Norm  group  report  (Report  No.  AFHRL 

TR-73-15).  Lowry  Air  Force  Base,  Colorado:  Air  Force  Human  Resources 

Laboratory,  Technical  Training  Division,  October  1973. 

ABSTRACT: 

Development  of  a new  student  critique  form  (SCF)  for  the  Air  Training  Comnand 
(ATC)  is  discussed.  The  study  had  four  objectives:  (a)  generate  more  items  for 

a previously-developed  prototype  scale  in  order  to  increase  its  reliability; 

(b)  administer  the  improved  SCF  to  large  numbers  of  officer,  NCOs  and  airmen 
enrolled  in  technical  courses;  (d)  factor  analyze  the  SCF  in  order  to  ensure  scale 
unidimensional ity;  and  (d)  norm  the  SCF  for  officers,  NCOs,  and  airmen.  Factor 
analyses  revealed  seven  unidimensional  scales:  (a)  Instructor  Competence; 

(b)  Study  Environment  and  Testing;  (c)  Specialty  Training;  (d)  Training  Impressions 
(e)  Classroom  Facilities  and  Environment;  (f)  Training  Devices  and  Audio-Visual 
Aids;  and  (g)  Training  Materials  Adequacy. 

A 69- item  SCF  was  administered  to  officers,  NCOs,  and  airmen  enrolled  in 
technical  training  courses  at  six  Air  Force  bases  (Chanute,  Keesler,  Sheppard, 
Lowry,  Lackland,  and  Goodfellow).  Scores  were  returned  for  each  rank.  Air  Force 
Specialty  Code  (AFSC),  and  Air  Force  base  where  the  training  center  was  located, 
as  individual  average  scores  and  as  scale  scores.  Scores  for  each  unidimensional 
scale  were  also  reported  by  rank,  AFSC,  and  base. 

Analyses  of  scores  returned  by  1,669  trainees  indicated:  (1)  attitudes  of 

trainees  toward  technical  training  were  "slightly"  positive;  (2)  the  most 
positive  attitudes  toward  technical  training  were  held  by  NCOs,  followed  by 
officers,  and  then  airmen;  (3)  since  statistically  significant  differences  existed 
among  scores  at  the  six  bases,  the  SCF  should  be  normed  separately  for  each  base; 
(4)  the  Instructor  Competence  scale  received  the  most  consistently-positive 
response  from  trainees;  and  (5)  the  Specialty  Training  Scale  received  the  most 
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consistently- negative  response  from  trainees,  which  indicated  a lack  of  assurance  — 
especially  among  airmen  — that  technical  training  was  adequately  preparing  them 
for  on-the-job  performance. 

The  next  phase  of  the  critique  improvement  program  will  be  development  and 
implementation  of  computer  software  for  managing  the  entire  student  critique 
program. 


Appendix  A contains  a sample  ATC  Form  736,  Student  Critique.  Appendix  B 
contains  the  improved  student  critique  form. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Print 

Conventional  and  individualized  instruction 
Internal  statistical  analysis 
Military  (Air  Force) 

5.1  Conduct  Internal  Evaluation 
Tool 
42 
18 
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REFERENCE  NUMBER:  5.2.1  CONDUCT  EXTERNAL  EVALUATION 

CITATION: 

Anastasio,  E.J.  An  evaluation  of  the  demonstrations  being  conducted  by  The 

University  of  Illinois  and  the  MITRE  Corporation  of  their  respective  computer- 
assisted  instructional  systems  (ETS  PR-72-19).  Princeton,  N.J.:  Educational 

Testing  Service,  1972.  (ERIC  Document  Reproduction  Service  No.  ED  072  070) 

ABSTRACT: 

Development  and  structure  of  a plan  for  evaluating  the  cost  performance  and 
educational  benefits  of  the  TICCIT  and  PLATO  projects  in  1971-72  are  reported. 
Included  with  the  report  are:  (a)  an  annotated  bibliography  of  documents  relating 

to  computer-based  education  in  general,  and  to  research  and  development  and 
evaulation  concerns  of  each  project;  (b)  Appendix  A,  Survey  of  Instructor 
Activities  and  Attitudes;  and  (c)  Appendix  B,  Attitudes  toward  Reading  Question- 
naire — Grade  2 and  Grades  4 and  6.  Due  to  schedule  slippages  in  each  project's 
development,  the  field-test  evaluation  has  been  repeatedly  postponed.  However,  as 
an  example  of  evaluation  procedure,  the  Educational  Testing  Service  (ETS)  report  is 
useful.  The  report  is  divided  into  three  sections. 

The  first  section  discusses  cost  analysis  of  PLATO  and  TICCIT  in  terms  of  five 
cost  categories:  (a)  nardware;  (b)  software;  (c)  courseware;  (d)  administration; 

and  (e)  incremental  site  (out-of-pocket).  A distinction  is  made  between  one-time 
costs  (developmental  and  implementation  expenses)  and  recurring  costs  (operating 
costs).  In  the  TICCIT  cost  analysis  section,  there  are  four  parts  to  hardware 
costs:  (a)  the  processor  facility;  (b)  the  communication  system;  (c)  the  terminal; 

and  (d)  audio-video  and  other,  including  personnel  costs.  There  are  three  parts 
to  software  costs:  (a)  application  software;  (b)  authoring  software;  and  (c) 

delivery  software.  Courseware  costs  are  figured  in  dollars-per-hour  for  course- 
ware development  by  the  following  specialists:  (a)  instructional  psychologist; 

(b)  instructional  design  technician;  (c)  empirical  design  technician;  (d)  packaging 
specialist;  (e)  author;  and  (f)  program  implementation.  Administration  costs  are 
determined  monthly  on  the  basis  of  the  ratio  of  each  course-development  cost  of 
the  total  monthly  developmental  costs.  Incremental  costs  include  expenses  for  modi 
fying  a site  for  instruction  (including  air  conditioning,  and  electrical  and 
facility  modification).  Cost  categories  for  the  PLATO  system  have  been  similarly 
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divided,  although  it  is  noted  that  the  very  different  natures  of  the  two  systems 
make  direct  comparisons  misleading. 

The  second  section  discusses  educational  analysis  of  the  two  systems  in  terms 
of:  (a)  achievement  and  attitude  measurement;  (b)  experimental  design;  and  (c) 

data  collection.  Student  achievement  would  be  measured  by  standardized  achievement 
tests,  with  addition  of  item-analysis  and  multiple-matrix  sampling  techniques  for 
obtaining  group  measurement.  Student  attitude  would  be  measured  by  five  standard- 
ized attitude  surveys  and  by  ETS-developed  items.  The  five  attitude  surveys  are: 

(a)  Student  Instructional  Report;  (b)  Comparative  Guidance  and  Placement  Program 
Student  Questionnaire;  (c)  College  Student  Questionnaire-Part  2;  (d)  College  and 
University  Environment  Scales;  and  (e)  Student  Reactions  to  College.  Several  of 
these  scales  were  still  under  development  in  1972.  Not  all  scales  would  be  completed 
by  any  one  student.  Faculty  attitudes  would  also  be  assessed.  Experimental  design 
considers  the  comparability  of  groups  taking  different  treatments,  comparability  of 
content  taken  by  taking  different  treatments,  obtaining  matched  pairs  of  students 
with  and  without  CAI,  and  appropriate  statistical  analyses  to  assess  CAI  program 
effectiveness.  Data  would  be  collected  from  each  project's  own  records  concerning 
system  cost  and  maintenance,  i.e.,  system  reliability.  In  addition,  system  per- 
formance variables  such  as  terminal  response  times,  queueing  effects,  ease  of  up- 
dating both  software  and  hardware,  and  user  satisfaction  with  the  system  would  be 
examined.  The  terminals  would  be  examined  for  comfort,  safety,  durability,  and 
special  environmental  or  wiring  requirements. 


The  third  section  covers  technical  analysis,  wl\ich  consisted  of  a number  of 
visits  by  ETS  to  TICCIT  and  PLATO  for  detailed  briefings  on  the  projected  hardware 
and  system  design.  No  conclusive  findings  are  given. 

DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 
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Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Civilian 
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REFERENCE  NUMBER:  5.3.1 

REVISE  SYSTEM 

CITATION: 

Havelock,  R.G.,  Havelock,  M.C.,  & Markowitz,  E.A.  Educational  innovation  in  the 

United  States.  Volume  I;  The  national  survey:  The  substance  and  the  process. 

Ann  Arbor:  The  University  of  Michigan,  Institute  for  Social  Research,  Center 

for  Research  on  Utilization  of  Scientific  Knowledge,  June  1973. 

ABSTRACT: 

This  report  is  an  assessment  of  educational  innovations  in  the  public  sector 
during  the  1970-71  school  year.  The  process  by  which  those  innovations  come  about 
or  continue  to  operate  is  discussed. 

Specific  findings,  which  may  be  of  some  interest,  are  as  follows:  (a)  of  353 
school  districts  sampled,  346  (98%)  reported  at  least  one  major  innovation  during 
the  year,  with  "major"  being  defined  as  a "substantial  reorientation  on  the  part  of 
the  staff,  a reallocation  of  resources,  or  adoption  of  new  practices,  programs,  or 
technology,"  (b)  individualized  instruction  and  team  teaching  were  the  most 
frequently  reported  innovations}  (c)  83%  of  the  innovations  were  successful;  (d) 
teachers  played  more  key  roles  in  the  innovations,  followed  by  assistant  super- 
intendents, and  then  principals;  (e)  the  greater  barrier  to  innovations  was 
confusion  among  staff  about  the  purpose  of  the  innovation;  and  (f)  school  district 
size  and  per-pupil  expenditure  positively  correlated  with  frequency  of  innovation, 
but  no  relationship  was  found  between  innovativeness  and  the  percent  of  graduates 
going  on  to  four  year  colleges,  the  pupil -teacher  ratio,  or  the  difficulty  in 
obtaining  financing  for  new  or  existing  programs. 

The  data  for  this  study  were  obtained  through  a very  detailed  but  well -formatted 
mail  questionnaire,  which  is  given  in  Appendix  A with  an  instruction  sheet.  The 
questionnaire  could  be  easily  adapted  for  obtaining  data  for  revising  an 
instructional  system.  The  three-page  questionnaire  is  designed  to  gather  data: 

(a)  describing  the  nature  of  the  innovation  and  how  it  came  about;  (b)  how  much 
emphasis  was  given  to  several  key  innovation  procedures;  (c)  what  barriers  to 
innovations  existed;  (d)  what  institutional  changes  in  both  the  local  school  and  school 
district  resulted  from  the  innovation;  (e)  what  internal  and  external  resources 
were  used  to  implement  the  innovation;  (f)  how  often  and  to  what  extent  the 
innovation  was  used;  and  (g)  specific  characteristics  of  the  school  district, 
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Including  information  on  policies  and  procedures,  system  size  and  staffing,  and 
frequency  of  teacher  strikes  and  student-community  group  protests. 


DATA  SUMMARY: 

Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Print 

Conventional  and  individualized  instruction 

None 

Civilian 

5.3  Revise  system 
Tool 

234+  (Report),  3 (Questionnaire) 

24  (Report) 
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REFERENCE  NUMBER:  5.3.2 

REVISE  SYSTEM 

CITATION: 

Haverland,  E.  M.  Transfer  and  use  of  training  technology:  A model  for  matching 

training  approaches  with  training  settings  (HumRRO  Tech.  Rep.  74-24). 

Alexandria,  Va.:  Human  Resources  Research  Organization,  October  1974. 

(NTIS  Document  Reproduction  Service  No.  AD  A005  816) 

ABSTRACT: 

A model  for  evaluating  training  approaches  or  innovations  in  terms  of 
requirements,  resources,  and  constraints  of  specific  training  settings  was 
developed  and  field-tested.  The  model  contains  two  parallel  sets  of  questions. 

The  first  set  of  questions  concerns  the  characteristics  of  the  training  approach 
under  consideration.  The  second  set  of  questions  concerns  the  requirements, 
resources,  and  constraints  in  the  specific  training  setting.  This  report 
discusses  the  development  of  the  preliminary  version  of  the  model,  its  field 
evaluation  by  Air  Force  training  personnel,  and  its  subsequent  revision.  A 
stand-alone  version  of  the  model  with  supporting  material  is  given  in  the  text. 
Relationship  of  the  model  to  Air  Force  procedures  for  Instructional  System 
Development  (ISO),  to  the  Rand  Corporation's  design  system  for  instructional  sys- 
tems (MODIA),  and  to  the  Air  Force's  Advanced  Instructional  System  (AIS)  is 
discussed.  The  application  of  the  model  to  several  different  training  approaches 
and  setting  is  discussed. 

With  matching,  parallel  sets  of  questions,  the  model  operationally  covers  the 
following  components  of  instructional  systems:  (a)  resources  required  and 

resources  available;  (b)  instructional  design  and  management;  (c)  training  content 
characteristics;  (d)  instructional  personnel;  and  (e)  student  characteristics. 

The  model  was  applied  to  the  training  approach  called  peer  instruction  used 
in  the  Law  Enforcement  Specialist  course  (3ABR81230)  at  Lackland  Air  Force  Base. 
Information  for  the  model  came  from  the  Course  Chart  and  the  Plan  of  Instruction 
for  the  course.  General  information  was  gathered  during  a site  visit  to  the 
School  of  Applied  Aerospace  Sciences  at  Lackland  Air  Force  Base. 
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Because  required  information  could  not  be  obtained  from  this  course,  the 
application  of  the  model  was  unsatisfactory,  but  plans  were  made  to  apply  the 
model  to  another  training  setting  in  close  collaboration  with  Air  Force  personnel. 
Critical  data  which  could  not  be  gathered  during  the  original  tryout  included: 

(a)  the  objectives  of  persons  working  in  the  training  setting  who  were  responsible 
for  its  efficient  and  effective  operation;  (b)  problems  existing  within  the  train- 
ing setting;  (c)  availability  of  space,  equipment,  and  support  services;  (d)  lead 
times  for  course  modifications;  (e)  school  operating  policies;  (f)  student  char- 
acteristics; (g)  availability  of  funds  in  the  training  setting;  and  (h)  character- 
istics of  the  training  content.  It  was  concluded  that  the  model  could  not  be 
applied  very  successfully  to  a training  setting  at  a distance  and  with  documentation 
intended  for  other  uses.  In  the  proposed  on-site  application  of  the  model,  a more 
rigorous  evaluation  of  the  model  will  be  conducted. 

Results  of  the  original  application  of  the  model  to  the  peer  instruction 
training  approach  are  given  in  Appendix  A. 


DATA  SUMMARY: 


Media: 

Application: 

Evaluation: 

Environment: 

Key  number: 

Authoring  aid  type: 
Number  of  pages: 
Number  of  references: 


Print 

Peer  instruction 
Field  test 

Military  (Air  Force) 

5.3  Revise  System 

Tool 

78 

24 
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